2024-09-17T15:35:23.440 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T15:35:23.459 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T15:35:23.550 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989 branch: wip-bharath10-testing-2024-09-17-1348 description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete} email: skanta@redhat.com first_in_suite: false job_id: '7908989' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath10-testing-2024-09-17-1348 ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2: false bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - slow request - overall HEALTH_ mon_bind_msgr2: false sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath10-testing-2024-09-17-1348 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mon.d - mon.e - osd.0 - osd.1 - osd.2 - - mon.f - mon.g - mon.h - mon.i - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 3421 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath10-testing-2024-09-17-1348 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKd61u5bTt7mzfpvYgDLEtHvBDLQTVqgMl5Ru2s78Rs5f7z0cs201ugVHdSox0HRtwVjXPRVNYFp5ebaLhS9HFw= smithi100.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnbv4lSTnyEFOtpCak4U6GM24J+nK0hAZY+rZviapQ3cNKYK4+RP0/lSf43uC1XqkqjjPfjcKzTL08d+IxINn0= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 thrash_many: true - exec: client.0: - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_15:01:33 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T15:35:23.550 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa; will attempt to use it 2024-09-17T15:35:23.551 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks 2024-09-17T15:35:23.551 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T15:35:23.552 INFO:teuthology.task.internal:Checking packages... 2024-09-17T15:35:23.574 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1889a165b5128787ec693450a3d1c5df5d6dc203' 2024-09-17T15:35:23.574 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T15:35:23.574 INFO:teuthology.packaging:ref: None 2024-09-17T15:35:23.574 INFO:teuthology.packaging:tag: None 2024-09-17T15:35:23.574 INFO:teuthology.packaging:branch: wip-bharath10-testing-2024-09-17-1348 2024-09-17T15:35:23.574 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:35:23.574 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath10-testing-2024-09-17-1348 2024-09-17T15:35:23.710 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4975.g1889a165 2024-09-17T15:35:23.712 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T15:35:23.719 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T15:35:23.719 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T15:35:23.754 INFO:teuthology.task.internal:Saving configuration 2024-09-17T15:35:23.769 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T15:35:23.780 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T15:35:23.804 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 15:30:19.658130', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKd61u5bTt7mzfpvYgDLEtHvBDLQTVqgMl5Ru2s78Rs5f7z0cs201ugVHdSox0HRtwVjXPRVNYFp5ebaLhS9HFw='} 2024-09-17T15:35:23.827 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 15:30:19.656087', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnbv4lSTnyEFOtpCak4U6GM24J+nK0hAZY+rZviapQ3cNKYK4+RP0/lSf43uC1XqkqjjPfjcKzTL08d+IxINn0='} 2024-09-17T15:35:23.827 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T15:35:23.834 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2024-09-17T15:35:23.834 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-17T15:35:23.834 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T15:35:23.906 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1c9631bbe0>, signals=[15]) 2024-09-17T15:35:23.906 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T15:35:23.912 INFO:teuthology.task.internal:Opening connections... 2024-09-17T15:35:23.912 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-09-17T15:35:23.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:35:23.990 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T15:35:23.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:35:24.073 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T15:35:24.079 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-09-17T15:35:24.097 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-09-17T15:35:24.097 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-09-17T15:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-09-17T15:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-09-17T15:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-09-17T15:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-09-17T15:35:24.154 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-09-17T15:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-09-17T15:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T15:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-09-17T15:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-09-17T15:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T15:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-09-17T15:35:24.155 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T15:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T15:35:24.156 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T15:35:24.156 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-09-17T15:35:24.174 DEBUG:teuthology.orchestra.run.smithi100:> uname -m 2024-09-17T15:35:24.192 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2024-09-17T15:35:24.193 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release 2024-09-17T15:35:24.250 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream" 2024-09-17T15:35:24.250 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="9" 2024-09-17T15:35:24.250 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2024-09-17T15:35:24.250 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2024-09-17T15:35:24.250 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="9" 2024-09-17T15:35:24.250 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el9" 2024-09-17T15:35:24.250 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T15:35:24.250 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2024-09-17T15:35:24.250 INFO:teuthology.orchestra.run.smithi100.stdout:LOGO="fedora-logo-icon" 2024-09-17T15:35:24.250 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T15:35:24.250 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/" 2024-09-17T15:35:24.251 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T15:35:24.251 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T15:35:24.251 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T15:35:24.251 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2024-09-17T15:35:24.274 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T15:35:24.284 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T15:35:24.320 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T15:35:24.320 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-09-17T15:35:24.322 DEBUG:teuthology.orchestra.run.smithi100:> test '!' -e /home/ubuntu/cephtest 2024-09-17T15:35:24.338 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T15:35:24.345 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T15:35:24.345 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-09-17T15:35:24.379 DEBUG:teuthology.orchestra.run.smithi100:> test -z $(ls -A /var/lib/ceph) 2024-09-17T15:35:24.488 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T15:35:24.537 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T15:35:24.613 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T15:35:24.613 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.f': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T15:35:24.613 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T15:35:24.614 DEBUG:teuthology.orchestra.run.smithi100:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T15:35:24.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:35:24.631 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-09-17T15:35:24.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:35:24.632 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-09-17T15:35:24.688 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-503.el9.x86_64 2024-09-17T15:35:24.688 INFO:teuthology.task.kernel:Running kernel on smithi100: 5.14.0-503.el9.x86_64 2024-09-17T15:35:24.688 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum install -y kernel 2024-09-17T15:35:24.689 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-503.el9.x86_64 2024-09-17T15:35:24.690 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-503.el9.x86_64 2024-09-17T15:35:24.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-09-17T15:35:28.216 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.3 MB 00:01 2024-09-17T15:35:28.258 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.3 MB 00:01 2024-09-17T15:35:33.733 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-17T15:35:39.109 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.5 MB 00:01 2024-09-17T15:35:40.924 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 1.8 MB/s | 20 MB 00:11 2024-09-17T15:35:41.493 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 19 kB 00:01 2024-09-17T15:35:42.297 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-17T15:35:45.442 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-17T15:35:46.967 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 19 kB 00:00 2024-09-17T15:35:47.721 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-17T15:35:47.736 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-17T15:35:49.601 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T15:35:49.601 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T15:35:49.696 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:35:49.703 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:35:49.703 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-17T15:35:49.703 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:35:49.703 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:35:49.703 INFO:teuthology.orchestra.run.smithi100.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T15:35:49.703 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T15:35:49.704 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T15:35:49.704 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T15:35:49.704 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T15:35:49.704 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:35:49.704 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:35:49.704 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:35:49.704 INFO:teuthology.orchestra.run.smithi100.stdout:Install 4 Packages 2024-09-17T15:35:49.704 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:35:49.705 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 79 M 2024-09-17T15:35:49.705 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 126 M 2024-09-17T15:35:49.705 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:35:49.896 INFO:teuthology.orchestra.run.smithi100.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 394 kB/s | 36 kB 00:00 2024-09-17T15:35:50.280 INFO:teuthology.orchestra.run.smithi100.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-17T15:35:50.655 INFO:teuthology.orchestra.run.smithi100.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 38 MB/s | 29 MB 00:00 2024-09-17T15:35:50.955 INFO:teuthology.orchestra.run.smithi100.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 30 MB/s | 35 MB 00:01 2024-09-17T15:35:50.955 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:35:50.955 INFO:teuthology.orchestra.run.smithi100.stdout:Total 63 MB/s | 79 MB 00:01 2024-09-17T15:35:51.641 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:35:51.984 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:35:51.985 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:35:52.951 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:35:52.952 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:35:53.294 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-17T15:35:55.292 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T15:35:55.292 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T15:35:55.388 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:35:55.394 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:35:55.396 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:35:55.396 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-09-17T15:35:55.396 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:35:55.396 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:35:55.397 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T15:35:55.397 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-17T15:35:55.397 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T15:35:55.397 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T15:35:55.397 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T15:35:55.397 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:35:55.397 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:35:55.397 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:35:55.397 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-09-17T15:35:55.397 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:35:55.398 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 79 M 2024-09-17T15:35:55.398 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 126 M 2024-09-17T15:35:55.398 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:35:55.545 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 283 kB/s | 36 kB 00:00 2024-09-17T15:35:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:35:56.264 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:35:56.737 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 12 MB/s | 16 MB 00:01 2024-09-17T15:35:57.421 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 15 MB/s | 29 MB 00:01 2024-09-17T15:35:58.162 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 13 MB/s | 35 MB 00:02 2024-09-17T15:35:58.163 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:35:58.163 INFO:teuthology.orchestra.run.smithi064.stdout:Total 28 MB/s | 79 MB 00:02 2024-09-17T15:35:58.870 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:35:59.019 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:35:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:35:59.243 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:35:59.243 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:36:00.250 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:36:00.250 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:36:02.795 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:36:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:36:03.529 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:36:04.007 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:36:04.277 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:36:06.272 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:36:06.397 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:36:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:36:11.522 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:36:11.564 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:36:16.707 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:36:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:36:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:36:47.470 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:36:47.470 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:36:47.470 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:36:47.470 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:36:48.273 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:36:48.273 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:36:48.273 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:36:48.273 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T15:36:48.273 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T15:36:48.273 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T15:36:48.274 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T15:36:48.274 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:36:48.274 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:36:48.614 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:36:48.708 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:36:48.709 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:36:48.709 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T15:36:48.709 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T15:36:48.709 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T15:36:48.709 INFO:teuthology.task.kernel:Installing distro kernel on mon.f... 2024-09-17T15:36:48.709 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi100.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T15:36:48.709 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:36:48.769 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:36:48.769 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:36:48.769 DEBUG:teuthology.orchestra.run.smithi100:> sudo rpm -qi grub2-tools 2024-09-17T15:36:48.811 INFO:teuthology.orchestra.run.smithi100.stdout:Name : grub2-tools 2024-09-17T15:36:48.811 INFO:teuthology.orchestra.run.smithi100.stdout:Epoch : 1 2024-09-17T15:36:48.811 INFO:teuthology.orchestra.run.smithi100.stdout:Version : 2.06 2024-09-17T15:36:48.811 INFO:teuthology.orchestra.run.smithi100.stdout:Release : 82.el9 2024-09-17T15:36:48.811 INFO:teuthology.orchestra.run.smithi100.stdout:Architecture: x86_64 2024-09-17T15:36:48.811 INFO:teuthology.orchestra.run.smithi100.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T15:36:48.811 INFO:teuthology.orchestra.run.smithi100.stdout:Group : Unspecified 2024-09-17T15:36:48.811 INFO:teuthology.orchestra.run.smithi100.stdout:Size : 8274670 2024-09-17T15:36:48.812 INFO:teuthology.orchestra.run.smithi100.stdout:License : GPLv3+ 2024-09-17T15:36:48.812 INFO:teuthology.orchestra.run.smithi100.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T15:36:48.812 INFO:teuthology.orchestra.run.smithi100.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T15:36:48.812 INFO:teuthology.orchestra.run.smithi100.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T15:36:48.812 INFO:teuthology.orchestra.run.smithi100.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T15:36:48.812 INFO:teuthology.orchestra.run.smithi100.stdout:Packager : builder@centos.org 2024-09-17T15:36:48.812 INFO:teuthology.orchestra.run.smithi100.stdout:Vendor : CentOS 2024-09-17T15:36:48.812 INFO:teuthology.orchestra.run.smithi100.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T15:36:48.812 INFO:teuthology.orchestra.run.smithi100.stdout:Summary : Support tools for GRUB. 2024-09-17T15:36:48.812 INFO:teuthology.orchestra.run.smithi100.stdout:Description : 2024-09-17T15:36:48.812 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:36:48.812 INFO:teuthology.orchestra.run.smithi100.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T15:36:48.812 INFO:teuthology.orchestra.run.smithi100.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T15:36:48.813 INFO:teuthology.orchestra.run.smithi100.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T15:36:48.813 INFO:teuthology.orchestra.run.smithi100.stdout:hardware devices. 2024-09-17T15:36:48.813 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:36:48.813 INFO:teuthology.orchestra.run.smithi100.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T15:36:48.814 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T15:36:48.814 INFO:teuthology.task.kernel:Updating grub on smithi100 to boot 5.14.0-508.el9.x86_64 2024-09-17T15:36:48.814 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T15:36:49.441 INFO:teuthology.orchestra.run.smithi100.stderr:Generating grub configuration file ... 2024-09-17T15:36:50.681 INFO:teuthology.orchestra.run.smithi100.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T15:36:50.721 INFO:teuthology.orchestra.run.smithi100.stderr:done 2024-09-17T15:36:50.723 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-09-17T15:36:50.739 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.GFqRCtzKpR 2024-09-17T15:36:50.739 DEBUG:teuthology.orchestra.run.smithi100:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GFqRCtzKpR 2024-09-17T15:36:50.806 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.GFqRCtzKpR 2024-09-17T15:36:50.955 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.GFqRCtzKpR is 6KB 2024-09-17T15:36:51.004 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.GFqRCtzKpR 2024-09-17T15:36:51.018 DEBUG:teuthology.orchestra.run.smithi100:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T15:36:51.081 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T15:36:51.081 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T15:36:51.081 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T15:36:51.081 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T15:36:51.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T15:36:51.234 DEBUG:teuthology.orchestra.run.smithi100:> sudo shutdown -r now 2024-09-17T15:36:55.871 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:36:55.914 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:36:56.173 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:36:56.173 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:36:56.173 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:36:56.173 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:36:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:36:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:36:56.988 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:36:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T15:36:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T15:36:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T15:36:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T15:36:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:36:56.989 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:36:57.346 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:36:57.436 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:36:57.436 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:36:57.436 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T15:36:57.437 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T15:36:57.437 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T15:36:57.437 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T15:36:57.437 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T15:36:57.437 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:36:57.496 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:36:57.496 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:36:57.496 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2024-09-17T15:36:57.535 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2024-09-17T15:36:57.536 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2024-09-17T15:36:57.536 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.06 2024-09-17T15:36:57.536 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 82.el9 2024-09-17T15:36:57.536 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2024-09-17T15:36:57.536 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T15:36:57.536 INFO:teuthology.orchestra.run.smithi064.stdout:Group : Unspecified 2024-09-17T15:36:57.536 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 8274670 2024-09-17T15:36:57.536 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2024-09-17T15:36:57.536 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T15:36:57.536 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T15:36:57.536 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T15:36:57.537 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T15:36:57.537 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : builder@centos.org 2024-09-17T15:36:57.537 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : CentOS 2024-09-17T15:36:57.537 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T15:36:57.537 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2024-09-17T15:36:57.537 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2024-09-17T15:36:57.537 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:36:57.537 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T15:36:57.537 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T15:36:57.537 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T15:36:57.537 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2024-09-17T15:36:57.537 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:36:57.537 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T15:36:57.538 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T15:36:57.538 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 5.14.0-508.el9.x86_64 2024-09-17T15:36:57.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T15:36:58.160 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-09-17T15:36:59.420 INFO:teuthology.orchestra.run.smithi064.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T15:36:59.458 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-09-17T15:36:59.461 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-09-17T15:36:59.476 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.teg58LYvip 2024-09-17T15:36:59.476 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.teg58LYvip 2024-09-17T15:36:59.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.teg58LYvip 2024-09-17T15:36:59.682 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.teg58LYvip is 6KB 2024-09-17T15:36:59.730 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.teg58LYvip 2024-09-17T15:36:59.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T15:36:59.807 INFO:teuthology.orchestra.run.smithi064.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T15:36:59.807 INFO:teuthology.orchestra.run.smithi064.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T15:36:59.807 INFO:teuthology.orchestra.run.smithi064.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T15:36:59.807 INFO:teuthology.orchestra.run.smithi064.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T15:36:59.809 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T15:36:59.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-09-17T15:37:21.257 INFO:teuthology.task.kernel:Checking client mon.f for new kernel version... 2024-09-17T15:37:21.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-17T15:37:21.259 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:37:29.977 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T15:37:29.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-17T15:37:29.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:38:04.477 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-09-17T15:38:13.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-17T15:38:13.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:38:21.269 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T15:38:30.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-17T15:38:30.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:38:30.599 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-09-17T15:38:31.033 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-17T15:38:31.033 INFO:teuthology.task.kernel:Checking kernel version of mon.f, want "5.14.0-508.el9.x86_64"... 2024-09-17T15:38:31.033 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-09-17T15:38:31.048 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-508.el9.x86_64 2024-09-17T15:38:31.048 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T15:38:31.048 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T15:38:31.048 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T15:38:31.869 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-09-17T15:38:32.050 INFO:teuthology.task.kernel:Enabling kdb on mon.f... 2024-09-17T15:38:32.050 DEBUG:teuthology.orchestra.run.smithi100:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T15:38:32.144 INFO:teuthology.orchestra.run.smithi100.stdout:ttyS1 2024-09-17T15:38:32.176 DEBUG:teuthology.parallel:result is None 2024-09-17T15:38:43.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-17T15:38:43.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:38:44.257 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-09-17T15:38:44.632 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-17T15:38:44.632 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T15:38:44.632 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-09-17T15:38:44.648 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-508.el9.x86_64 2024-09-17T15:38:44.648 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T15:38:44.648 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T15:38:44.648 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T15:38:45.650 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T15:38:45.650 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T15:38:45.744 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-09-17T15:38:45.772 DEBUG:teuthology.parallel:result is None 2024-09-17T15:38:45.772 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T15:38:45.784 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T15:38:45.785 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T15:38:45.787 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T15:38:45.808 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T15:38:45.817 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T15:38:45.853 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T15:38:45.854 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T15:38:45.856 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T15:38:45.883 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T15:38:45.890 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T15:38:45.891 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T15:38:45.917 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T15:38:45.948 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:38:45.954 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:38:45.960 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:38:45.965 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:38:45.967 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T15:38:45.975 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T15:38:45.975 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T15:38:46.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T15:38:46.042 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T15:38:46.052 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T15:38:46.052 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T15:38:46.085 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T15:38:46.102 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:38:46.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:38:46.237 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:38:46.325 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:38:46.379 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:38:46.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T15:38:46.448 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:38:46.493 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:38:46.544 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:38:46.630 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:38:46.671 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:38:46.671 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T15:38:46.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-09-17T15:38:46.741 DEBUG:teuthology.orchestra.run.smithi100:> sudo service rsyslog restart 2024-09-17T15:38:46.792 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T15:38:46.838 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T15:38:47.342 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T15:38:47.352 INFO:teuthology.task.internal:Starting timer... 2024-09-17T15:38:47.353 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T15:38:47.396 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T15:38:47.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-09-17T15:38:47.522 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-09-17T15:38:47.524 DEBUG:teuthology.orchestra.run.smithi100:> sudo service auditd rotate 2024-09-17T15:38:47.578 INFO:teuthology.orchestra.run.smithi100.stdout:Rotating logs: 2024-09-17T15:38:47.579 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T15:38:47.580 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-09-17T15:38:47.629 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-09-17T15:38:47.629 DEBUG:teuthology.orchestra.run.smithi100:> /usr/sbin/getenforce 2024-09-17T15:38:47.653 INFO:teuthology.orchestra.run.smithi100.stdout:Permissive 2024-09-17T15:38:47.653 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi100.front.sepia.ceph.com': 'permissive'} 2024-09-17T15:38:47.654 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T15:38:47.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:38:47.740 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T15:38:47.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:38:47.770 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T15:38:47.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-09-17T15:38:47.994 DEBUG:teuthology.orchestra.run.smithi100:> sudo /usr/sbin/setenforce permissive 2024-09-17T15:38:48.035 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T15:38:48.046 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T15:38:48.057 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T15:38:48.058 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi100.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T15:42:36.643 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi100.front.sepia.ceph.com')] 2024-09-17T15:42:36.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-17T15:42:36.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:42:36.731 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-09-17T15:42:36.816 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-17T15:42:36.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-17T15:42:36.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:42:36.893 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-09-17T15:42:36.976 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-17T15:42:36.977 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T15:42:36.986 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T15:42:36.987 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T15:42:36.987 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:42:36.990 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T15:42:36.990 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:42:37.018 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T15:42:37.033 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T15:42:37.049 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T15:42:37.058 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-09-17T15:42:37.063 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T15:42:37.070 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-09-17T15:42:37.085 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T15:42:37.087 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: ntpd: command not found 2024-09-17T15:42:37.098 INFO:teuthology.orchestra.run.smithi100.stdout:506 Cannot talk to daemon 2024-09-17T15:42:37.099 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T15:42:37.112 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T15:42:37.126 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T15:42:37.160 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-09-17T15:42:37.163 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T15:42:37.163 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-09-17T15:42:37.163 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:42:37.163 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:42:37.163 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:42:37.163 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:42:37.180 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-09-17T15:42:37.184 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T15:42:37.184 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-09-17T15:42:37.184 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:42:37.184 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:42:37.184 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:42:37.184 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:42:37.185 INFO:teuthology.run_tasks:Running task install... 2024-09-17T15:42:37.194 DEBUG:teuthology.task.install:project ceph 2024-09-17T15:42:37.194 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}} 2024-09-17T15:42:37.194 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'} 2024-09-17T15:42:37.194 INFO:teuthology.task.install:Using flavor: default 2024-09-17T15:42:37.200 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T15:42:37.200 INFO:teuthology.task.install:extra packages: [] 2024-09-17T15:42:37.200 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T15:42:37.201 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:42:37.203 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T15:42:37.203 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:42:37.365 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/ 2024-09-17T15:42:37.366 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4975.g1889a165 2024-09-17T15:42:37.371 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/ 2024-09-17T15:42:37.371 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4975.g1889a165 2024-09-17T15:42:37.488 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T15:42:37.488 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:42:37.488 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T15:42:37.494 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T15:42:37.494 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:42:37.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T15:42:37.514 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T15:42:37.515 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1889a165b5128787ec693450a3d1c5df5d6dc203/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T15:42:37.520 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T15:42:37.520 DEBUG:teuthology.orchestra.run.smithi100:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1889a165b5128787ec693450a3d1c5df5d6dc203/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T15:42:37.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T15:42:37.587 DEBUG:teuthology.orchestra.run.smithi100:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T15:42:37.659 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T15:42:37.661 DEBUG:teuthology.orchestra.run.smithi100:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T15:42:37.724 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-09-17T15:42:37.725 INFO:teuthology.orchestra.run.smithi100.stdout:check_obsoletes = 1 2024-09-17T15:42:37.726 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-09-17T15:42:37.727 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum clean all 2024-09-17T15:42:37.955 INFO:teuthology.orchestra.run.smithi100.stdout:45 files removed 2024-09-17T15:42:37.970 INFO:teuthology.orchestra.run.smithi064.stdout:45 files removed 2024-09-17T15:42:37.982 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-radosgw 2024-09-17T15:42:37.997 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw 2024-09-17T15:42:38.531 INFO:teuthology.orchestra.run.smithi100.stdout:ceph packages for x86_64 315 kB/s | 84 kB 00:00 2024-09-17T15:42:38.571 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 304 kB/s | 84 kB 00:00 2024-09-17T15:42:38.839 INFO:teuthology.orchestra.run.smithi100.stdout:ceph noarch packages 91 kB/s | 24 kB 00:00 2024-09-17T15:42:38.889 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 89 kB/s | 24 kB 00:00 2024-09-17T15:42:39.123 INFO:teuthology.orchestra.run.smithi100.stdout:ceph source packages 7.7 kB/s | 1.9 kB 00:00 2024-09-17T15:42:39.164 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-17T15:42:39.648 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.3 MB 00:00 2024-09-17T15:42:41.064 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.3 MB 00:01 2024-09-17T15:42:41.358 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 32 MB/s | 20 MB 00:00 2024-09-17T15:42:45.667 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-17T15:42:46.349 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 4.9 MB/s | 20 MB 00:04 2024-09-17T15:42:47.218 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 98 kB/s | 19 kB 00:00 2024-09-17T15:42:48.305 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-17T15:42:50.816 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-17T15:42:52.500 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 19 kB 00:00 2024-09-17T15:42:53.229 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 38 MB/s | 23 MB 00:00 2024-09-17T15:42:53.788 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-09-17T15:42:56.073 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:42:56.083 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:42:56.083 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T15:42:56.084 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:42:56.084 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:42:56.084 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 11 M 2024-09-17T15:42:56.084 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrading: 2024-09-17T15:42:56.084 INFO:teuthology.orchestra.run.smithi100.stdout: librados2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.5 M 2024-09-17T15:42:56.084 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.3 M 2024-09-17T15:42:56.084 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T15:42:56.084 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T15:42:56.084 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:42:56.084 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 22 M 2024-09-17T15:42:56.084 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux x86_64 2:19.3.0-4975.g1889a165.el9 ceph 25 k 2024-09-17T15:42:56.084 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T15:42:56.084 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T15:42:56.085 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T15:42:56.085 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 756 k 2024-09-17T15:42:56.085 INFO:teuthology.orchestra.run.smithi100.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T15:42:56.085 INFO:teuthology.orchestra.run.smithi100.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T15:42:56.085 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T15:42:56.085 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T15:42:56.085 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 508 k 2024-09-17T15:42:56.085 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T15:42:56.085 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:42:56.085 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T15:42:56.085 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T15:42:56.085 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T15:42:56.085 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 45 k 2024-09-17T15:42:56.086 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 143 k 2024-09-17T15:42:56.086 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs x86_64 2:19.3.0-4975.g1889a165.el9 ceph 161 k 2024-09-17T15:42:56.086 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T15:42:56.086 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados x86_64 2:19.3.0-4975.g1889a165.el9 ceph 321 k 2024-09-17T15:42:56.086 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 305 k 2024-09-17T15:42:56.086 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 99 k 2024-09-17T15:42:56.086 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T15:42:56.086 INFO:teuthology.orchestra.run.smithi100.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T15:42:56.086 INFO:teuthology.orchestra.run.smithi100.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T15:42:56.086 INFO:teuthology.orchestra.run.smithi100.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T15:42:56.086 INFO:teuthology.orchestra.run.smithi100.stdout:Installing weak dependencies: 2024-09-17T15:42:56.086 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T15:42:56.086 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:42:56.087 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:42:56.087 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:42:56.087 INFO:teuthology.orchestra.run.smithi100.stdout:Install 31 Packages 2024-09-17T15:42:56.087 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrade 2 Packages 2024-09-17T15:42:56.087 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:42:56.087 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 62 M 2024-09-17T15:42:56.087 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:42:56.915 INFO:teuthology.orchestra.run.smithi100.stdout:(1/33): ceph-base-19.3.0-4975.g1889a165.el9.x86 13 MB/s | 5.5 MB 00:00 2024-09-17T15:42:56.973 INFO:teuthology.orchestra.run.smithi100.stdout:(2/33): ceph-selinux-19.3.0-4975.g1889a165.el9. 434 kB/s | 25 kB 00:00 2024-09-17T15:42:57.064 INFO:teuthology.orchestra.run.smithi100.stdout:(3/33): libcephfs2-19.3.0-4975.g1889a165.el9.x8 8.1 MB/s | 756 kB 00:00 2024-09-17T15:42:57.140 INFO:teuthology.orchestra.run.smithi100.stdout:(4/33): libradosstriper1-19.3.0-4975.g1889a165. 6.6 MB/s | 508 kB 00:00 2024-09-17T15:42:57.457 INFO:teuthology.orchestra.run.smithi100.stdout:(5/33): librgw2-19.3.0-4975.g1889a165.el9.x86_6 17 MB/s | 5.5 MB 00:00 2024-09-17T15:42:57.590 INFO:teuthology.orchestra.run.smithi100.stdout:(6/33): ceph-radosgw-19.3.0-4975.g1889a165.el9. 9.8 MB/s | 11 MB 00:01 2024-09-17T15:42:57.607 INFO:teuthology.orchestra.run.smithi100.stdout:(7/33): python3-ceph-argparse-19.3.0-4975.g1889 300 kB/s | 45 kB 00:00 2024-09-17T15:42:57.641 INFO:teuthology.orchestra.run.smithi100.stdout:(8/33): python3-ceph-common-19.3.0-4975.g1889a1 2.8 MB/s | 143 kB 00:00 2024-09-17T15:42:57.666 INFO:teuthology.orchestra.run.smithi100.stdout:(9/33): python3-cephfs-19.3.0-4975.g1889a165.el 2.7 MB/s | 161 kB 00:00 2024-09-17T15:42:57.691 INFO:teuthology.orchestra.run.smithi100.stdout:(10/33): python3-rados-19.3.0-4975.g1889a165.el 6.3 MB/s | 321 kB 00:00 2024-09-17T15:42:57.717 INFO:teuthology.orchestra.run.smithi100.stdout:(11/33): python3-rbd-19.3.0-4975.g1889a165.el9. 6.0 MB/s | 305 kB 00:00 2024-09-17T15:42:57.742 INFO:teuthology.orchestra.run.smithi100.stdout:(12/33): python3-rgw-19.3.0-4975.g1889a165.el9. 1.9 MB/s | 99 kB 00:00 2024-09-17T15:42:57.951 INFO:teuthology.orchestra.run.smithi100.stdout:(13/33): ceph-common-19.3.0-4975.g1889a165.el9. 15 MB/s | 22 MB 00:01 2024-09-17T15:42:58.009 INFO:teuthology.orchestra.run.smithi100.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 366 kB/s | 107 kB 00:00 2024-09-17T15:42:58.059 INFO:teuthology.orchestra.run.smithi100.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 909 kB/s | 45 kB 00:00 2024-09-17T15:42:58.084 INFO:teuthology.orchestra.run.smithi100.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 498 kB/s | 171 kB 00:00 2024-09-17T15:42:58.110 INFO:teuthology.orchestra.run.smithi100.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-17T15:42:58.160 INFO:teuthology.orchestra.run.smithi100.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-09-17T15:42:58.186 INFO:teuthology.orchestra.run.smithi100.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 662 kB 00:00 2024-09-17T15:42:58.211 INFO:teuthology.orchestra.run.smithi100.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 420 kB/s | 42 kB 00:00 2024-09-17T15:42:58.236 INFO:teuthology.orchestra.run.smithi100.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 717 kB/s | 36 kB 00:00 2024-09-17T15:42:58.262 INFO:teuthology.orchestra.run.smithi100.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 938 kB/s | 47 kB 00:00 2024-09-17T15:42:58.287 INFO:teuthology.orchestra.run.smithi100.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-17T15:42:58.345 INFO:teuthology.orchestra.run.smithi100.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-17T15:42:58.371 INFO:teuthology.orchestra.run.smithi100.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-17T15:42:58.396 INFO:teuthology.orchestra.run.smithi100.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 275 kB/s | 26 kB 00:00 2024-09-17T15:42:58.421 INFO:teuthology.orchestra.run.smithi100.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T15:42:58.455 INFO:teuthology.orchestra.run.smithi100.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-17T15:42:58.505 INFO:teuthology.orchestra.run.smithi100.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-17T15:42:58.555 INFO:teuthology.orchestra.run.smithi100.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-17T15:42:58.639 INFO:teuthology.orchestra.run.smithi100.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-17T15:42:58.764 INFO:teuthology.orchestra.run.smithi100.stdout:(32/33): librados2-19.3.0-4975.g1889a165.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-17T15:42:58.823 INFO:teuthology.orchestra.run.smithi100.stdout:(33/33): librbd1-19.3.0-4975.g1889a165.el9.x86_ 12 MB/s | 3.3 MB 00:00 2024-09-17T15:42:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:42:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:Total 23 MB/s | 62 MB 00:02 2024-09-17T15:42:58.927 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T15:42:58.946 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:42:59.013 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:42:59.014 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:42:59.708 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:42:59.708 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:43:01.214 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:43:01.223 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-17T15:43:01.223 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:01.223 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:43:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 11 M 2024-09-17T15:43:01.223 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-09-17T15:43:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.5 M 2024-09-17T15:43:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.3 M 2024-09-17T15:43:01.223 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-17T15:43:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T15:43:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:43:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 22 M 2024-09-17T15:43:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.3.0-4975.g1889a165.el9 ceph 25 k 2024-09-17T15:43:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T15:43:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T15:43:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T15:43:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 756 k 2024-09-17T15:43:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T15:43:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T15:43:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T15:43:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T15:43:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 508 k 2024-09-17T15:43:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T15:43:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:43:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T15:43:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T15:43:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T15:43:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 45 k 2024-09-17T15:43:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 143 k 2024-09-17T15:43:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.3.0-4975.g1889a165.el9 ceph 161 k 2024-09-17T15:43:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T15:43:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.3.0-4975.g1889a165.el9 ceph 321 k 2024-09-17T15:43:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 305 k 2024-09-17T15:43:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 99 k 2024-09-17T15:43:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T15:43:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T15:43:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T15:43:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T15:43:01.226 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-09-17T15:43:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T15:43:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:01.226 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:43:01.226 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:01.226 INFO:teuthology.orchestra.run.smithi064.stdout:Install 31 Packages 2024-09-17T15:43:01.226 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-09-17T15:43:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:01.227 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 62 M 2024-09-17T15:43:01.227 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:43:01.233 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:43:01.314 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-17T15:43:01.344 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-17T15:43:01.369 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-17T15:43:01.392 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:43:01.415 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:43:01.562 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:43:01.615 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:43:02.025 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-17T15:43:02.088 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:43:02.171 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:43:02.259 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 8/35 2024-09-17T15:43:02.281 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:43:02.344 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:43:02.344 INFO:teuthology.orchestra.run.smithi064.stdout:(1/33): ceph-base-19.3.0-4975.g1889a165.el9.x86 8.5 MB/s | 5.5 MB 00:00 2024-09-17T15:43:02.381 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-17T15:43:02.395 INFO:teuthology.orchestra.run.smithi064.stdout:(2/33): ceph-selinux-19.3.0-4975.g1889a165.el9. 506 kB/s | 25 kB 00:00 2024-09-17T15:43:02.424 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 11/35 2024-09-17T15:43:02.486 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:43:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:43:02.512 INFO:teuthology.orchestra.run.smithi064.stdout:(3/33): ceph-radosgw-19.3.0-4975.g1889a165.el9. 13 MB/s | 11 MB 00:00 2024-09-17T15:43:02.545 INFO:teuthology.orchestra.run.smithi064.stdout:(4/33): libcephfs2-19.3.0-4975.g1889a165.el9.x8 4.9 MB/s | 756 kB 00:00 2024-09-17T15:43:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:43:02.578 INFO:teuthology.orchestra.run.smithi064.stdout:(5/33): libradosstriper1-19.3.0-4975.g1889a165. 7.5 MB/s | 508 kB 00:00 2024-09-17T15:43:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-17T15:43:02.646 INFO:teuthology.orchestra.run.smithi064.stdout:(6/33): python3-ceph-argparse-19.3.0-4975.g1889 678 kB/s | 45 kB 00:00 2024-09-17T15:43:02.722 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-17T15:43:02.729 INFO:teuthology.orchestra.run.smithi064.stdout:(7/33): python3-ceph-common-19.3.0-4975.g1889a1 1.7 MB/s | 143 kB 00:00 2024-09-17T15:43:02.764 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-17T15:43:02.811 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-17T15:43:02.813 INFO:teuthology.orchestra.run.smithi064.stdout:(8/33): librgw2-19.3.0-4975.g1889a165.el9.x86_6 21 MB/s | 5.5 MB 00:00 2024-09-17T15:43:02.839 INFO:teuthology.orchestra.run.smithi064.stdout:(9/33): python3-cephfs-19.3.0-4975.g1889a165.el 1.5 MB/s | 161 kB 00:00 2024-09-17T15:43:02.840 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-17T15:43:02.863 INFO:teuthology.orchestra.run.smithi064.stdout:(10/33): python3-rados-19.3.0-4975.g1889a165.el 6.3 MB/s | 321 kB 00:00 2024-09-17T15:43:02.869 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-17T15:43:02.889 INFO:teuthology.orchestra.run.smithi064.stdout:(11/33): python3-rbd-19.3.0-4975.g1889a165.el9. 6.0 MB/s | 305 kB 00:00 2024-09-17T15:43:02.902 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:43:02.916 INFO:teuthology.orchestra.run.smithi064.stdout:(12/33): python3-rgw-19.3.0-4975.g1889a165.el9. 1.8 MB/s | 99 kB 00:00 2024-09-17T15:43:02.940 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-17T15:43:03.125 INFO:teuthology.orchestra.run.smithi064.stdout:(13/33): ceph-common-19.3.0-4975.g1889a165.el9. 15 MB/s | 22 MB 00:01 2024-09-17T15:43:03.184 INFO:teuthology.orchestra.run.smithi064.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 363 kB/s | 107 kB 00:00 2024-09-17T15:43:03.265 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-17T15:43:03.292 INFO:teuthology.orchestra.run.smithi064.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 454 kB/s | 171 kB 00:00 2024-09-17T15:43:03.309 INFO:teuthology.orchestra.run.smithi064.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 361 kB/s | 45 kB 00:00 2024-09-17T15:43:03.315 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:43:03.334 INFO:teuthology.orchestra.run.smithi064.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 767 kB/s | 160 kB 00:00 2024-09-17T15:43:03.393 INFO:teuthology.orchestra.run.smithi064.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 723 kB/s | 42 kB 00:00 2024-09-17T15:43:03.407 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:43:03.435 INFO:teuthology.orchestra.run.smithi064.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-17T15:43:03.437 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 24/35 2024-09-17T15:43:03.468 INFO:teuthology.orchestra.run.smithi064.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-17T15:43:03.473 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-17T15:43:03.494 INFO:teuthology.orchestra.run.smithi064.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 612 kB/s | 36 kB 00:00 2024-09-17T15:43:03.527 INFO:teuthology.orchestra.run.smithi064.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-09-17T15:43:03.552 INFO:teuthology.orchestra.run.smithi064.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 567 kB/s | 47 kB 00:00 2024-09-17T15:43:03.628 INFO:teuthology.orchestra.run.smithi064.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-09-17T15:43:03.653 INFO:teuthology.orchestra.run.smithi064.stdout:(25/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.9 MB/s | 308 kB 00:00 2024-09-17T15:43:03.678 INFO:teuthology.orchestra.run.smithi064.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 967 kB/s | 49 kB 00:00 2024-09-17T15:43:03.712 INFO:teuthology.orchestra.run.smithi064.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-09-17T15:43:03.745 INFO:teuthology.orchestra.run.smithi064.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-17T15:43:03.779 INFO:teuthology.orchestra.run.smithi064.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-17T15:43:03.838 INFO:teuthology.orchestra.run.smithi064.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-17T15:43:03.913 INFO:teuthology.orchestra.run.smithi064.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-17T15:43:03.973 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-17T15:43:04.002 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:43:04.055 INFO:teuthology.orchestra.run.smithi064.stdout:(32/33): librados2-19.3.0-4975.g1889a165.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-17T15:43:04.062 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:43:04.122 INFO:teuthology.orchestra.run.smithi064.stdout:(33/33): librbd1-19.3.0-4975.g1889a165.el9.x86_ 11 MB/s | 3.3 MB 00:00 2024-09-17T15:43:04.127 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:43:04.127 INFO:teuthology.orchestra.run.smithi064.stdout:Total 21 MB/s | 62 MB 00:02 2024-09-17T15:43:04.134 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 28/35 2024-09-17T15:43:04.162 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 29/35 2024-09-17T15:43:04.243 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:43:04.285 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:43:04.285 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:43:04.965 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:43:04.966 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:43:06.387 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:43:06.425 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:43:06.526 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:43:06.607 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-17T15:43:06.635 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-17T15:43:06.661 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-17T15:43:06.683 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:43:06.709 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:43:06.904 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:43:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:43:06.989 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:43:07.020 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:43:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:43:07.105 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T15:43:07.105 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T15:43:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:07.149 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:43:07.318 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-17T15:43:07.420 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:43:07.513 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:43:07.604 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 8/35 2024-09-17T15:43:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:43:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:43:07.747 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-17T15:43:07.791 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 11/35 2024-09-17T15:43:07.862 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:43:07.891 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:43:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:43:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-17T15:43:08.118 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-17T15:43:08.158 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-17T15:43:08.198 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-17T15:43:08.230 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-17T15:43:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-17T15:43:08.300 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:43:08.339 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-17T15:43:08.675 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-17T15:43:08.722 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:43:08.814 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:43:08.844 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 24/35 2024-09-17T15:43:08.881 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-17T15:43:09.408 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-17T15:43:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:43:09.489 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:43:09.569 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 28/35 2024-09-17T15:43:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 29/35 2024-09-17T15:43:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:43:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:43:12.888 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:43:12.927 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:43:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:43:13.019 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T15:43:13.019 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T15:43:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:43:17.131 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:43:17.132 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /sys 2024-09-17T15:43:17.132 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /proc 2024-09-17T15:43:17.132 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /mnt 2024-09-17T15:43:17.132 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /var/tmp 2024-09-17T15:43:17.132 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /home 2024-09-17T15:43:17.132 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /root 2024-09-17T15:43:17.132 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /tmp 2024-09-17T15:43:17.132 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:43:18.158 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:43:18.158 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:43:18.158 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T15:43:18.158 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:43:18.159 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:43:18.159 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:18.160 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:43:18.253 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:43:18.253 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:43:19.508 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:43:19.509 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 1/35 2024-09-17T15:43:19.509 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2/35 2024-09-17T15:43:19.509 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 3/35 2024-09-17T15:43:19.509 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 4/35 2024-09-17T15:43:19.509 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 5/35 2024-09-17T15:43:19.509 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 6/35 2024-09-17T15:43:19.509 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:43:19.509 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 8/35 2024-09-17T15:43:19.510 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 9/35 2024-09-17T15:43:19.510 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 10/35 2024-09-17T15:43:19.510 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 11/35 2024-09-17T15:43:19.510 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:43:19.510 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 13/35 2024-09-17T15:43:19.510 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-17T15:43:19.510 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-17T15:43:19.510 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-17T15:43:19.510 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-17T15:43:19.510 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-17T15:43:19.510 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-17T15:43:19.511 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:43:19.511 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-17T15:43:19.511 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-17T15:43:19.511 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-17T15:43:19.511 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-17T15:43:19.511 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-17T15:43:19.511 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-17T15:43:19.511 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-17T15:43:19.511 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-17T15:43:19.511 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-17T15:43:19.511 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-17T15:43:19.511 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-17T15:43:19.511 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:43:19.512 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-17T15:43:19.512 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 34/35 2024-09-17T15:43:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:43:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:20.436 INFO:teuthology.orchestra.run.smithi100.stdout:Upgraded: 2024-09-17T15:43:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.437 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:43:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T15:43:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T15:43:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T15:43:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T15:43:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T15:43:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T15:43:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T15:43:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T15:43:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:20.439 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:43:20.779 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-test 2024-09-17T15:43:21.364 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:28 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:43:21.499 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:43:21.500 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T15:43:21.500 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:21.500 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:43:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test x86_64 2:19.3.0-4975.g1889a165.el9 ceph 46 M 2024-09-17T15:43:21.500 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T15:43:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite x86_64 2:19.3.0-4975.g1889a165.el9 ceph 162 k 2024-09-17T15:43:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T15:43:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T15:43:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:21.501 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:43:21.501 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:21.501 INFO:teuthology.orchestra.run.smithi100.stdout:Install 4 Packages 2024-09-17T15:43:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:21.502 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 46 M 2024-09-17T15:43:21.502 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 201 M 2024-09-17T15:43:21.502 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:43:21.840 INFO:teuthology.orchestra.run.smithi100.stdout:(1/4): libcephsqlite-19.3.0-4975.g1889a165.el9. 775 kB/s | 162 kB 00:00 2024-09-17T15:43:22.207 INFO:teuthology.orchestra.run.smithi100.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 174 kB/s | 64 kB 00:00 2024-09-17T15:43:22.266 INFO:teuthology.orchestra.run.smithi100.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 480 kB/s | 304 kB 00:00 2024-09-17T15:43:23.058 INFO:teuthology.orchestra.run.smithi100.stdout:(4/4): ceph-test-19.3.0-4975.g1889a165.el9.x86_ 32 MB/s | 46 MB 00:01 2024-09-17T15:43:23.060 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:43:23.060 INFO:teuthology.orchestra.run.smithi100.stdout:Total 30 MB/s | 46 MB 00:01 2024-09-17T15:43:23.089 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:43:23.109 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:43:23.109 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:43:23.489 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:43:23.490 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:43:23.693 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:43:23.693 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-09-17T15:43:23.693 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-09-17T15:43:23.693 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-09-17T15:43:23.693 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-09-17T15:43:23.693 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-09-17T15:43:23.694 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-09-17T15:43:23.694 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-09-17T15:43:23.694 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:43:24.066 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T15:43:24.121 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T15:43:24.151 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:43:24.739 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:43:24.765 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:43:24.765 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:43:24.765 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T15:43:24.765 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:43:24.765 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:43:24.765 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:24.767 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:43:24.849 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:43:24.849 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:43:26.194 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:43:26.194 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 1/35 2024-09-17T15:43:26.194 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2/35 2024-09-17T15:43:26.194 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 3/35 2024-09-17T15:43:26.194 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 4/35 2024-09-17T15:43:26.194 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 5/35 2024-09-17T15:43:26.194 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 6/35 2024-09-17T15:43:26.194 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:43:26.194 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 8/35 2024-09-17T15:43:26.194 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 9/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 10/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 11/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 13/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-17T15:43:26.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-17T15:43:26.196 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-17T15:43:26.196 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-17T15:43:26.196 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-17T15:43:26.196 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-17T15:43:26.196 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-17T15:43:26.196 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-17T15:43:26.196 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-17T15:43:26.196 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:43:26.196 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-17T15:43:26.197 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 34/35 2024-09-17T15:43:27.198 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:43:27.198 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:27.198 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-09-17T15:43:27.198 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.198 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.198 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:43:27.198 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T15:43:27.199 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.199 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.199 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.199 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.199 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T15:43:27.199 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T15:43:27.199 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T15:43:27.199 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.199 INFO:teuthology.orchestra.run.smithi064.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T15:43:27.199 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T15:43:27.199 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T15:43:27.199 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T15:43:27.199 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.200 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T15:43:27.200 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.200 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T15:43:27.200 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T15:43:27.200 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T15:43:27.200 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.200 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.200 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.200 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T15:43:27.200 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.200 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.200 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:27.200 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T15:43:27.201 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T15:43:27.201 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T15:43:27.201 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T15:43:27.201 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T15:43:27.201 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:27.201 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:43:27.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test 2024-09-17T15:43:28.176 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:30 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:43:28.323 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:43:28.324 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:28.324 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-17T15:43:28.324 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:28.324 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:43:28.324 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.3.0-4975.g1889a165.el9 ceph 46 M 2024-09-17T15:43:28.324 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-17T15:43:28.324 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.3.0-4975.g1889a165.el9 ceph 162 k 2024-09-17T15:43:28.324 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T15:43:28.325 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T15:43:28.325 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:28.325 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:43:28.325 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:28.325 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-09-17T15:43:28.325 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:28.326 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 46 M 2024-09-17T15:43:28.326 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 201 M 2024-09-17T15:43:28.326 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:43:28.578 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:43:28.604 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-17T15:43:28.638 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-09-17T15:43:28.675 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:43:28.821 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): libcephsqlite-19.3.0-4975.g1889a165.el9. 444 kB/s | 162 kB 00:00 2024-09-17T15:43:30.955 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): ceph-test-19.3.0-4975.g1889a165.el9.x86_ 18 MB/s | 46 MB 00:02 2024-09-17T15:43:30.956 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:43:30.957 INFO:teuthology.orchestra.run.smithi064.stdout:Total 17 MB/s | 46 MB 00:02 2024-09-17T15:43:30.985 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:43:31.002 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:43:31.003 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 1/4 2024-09-17T15:43:31.003 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2/4 2024-09-17T15:43:31.003 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T15:43:31.008 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:43:31.008 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:43:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T15:43:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:31.317 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:43:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T15:43:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T15:43:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:31.318 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:43:31.403 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:43:31.403 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:43:31.623 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph 2024-09-17T15:43:31.930 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:43:32.114 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T15:43:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T15:43:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:43:32.233 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:39 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:43:32.384 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:43:32.387 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:32.387 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T15:43:32.387 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:32.387 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:43:32.387 INFO:teuthology.orchestra.run.smithi100.stdout: ceph x86_64 2:19.3.0-4975.g1889a165.el9 ceph 6.6 k 2024-09-17T15:43:32.387 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T15:43:32.387 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds x86_64 2:19.3.0-4975.g1889a165.el9 ceph 2.4 M 2024-09-17T15:43:32.388 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr x86_64 2:19.3.0-4975.g1889a165.el9 ceph 1.5 M 2024-09-17T15:43:32.388 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 291 k 2024-09-17T15:43:32.388 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon x86_64 2:19.3.0-4975.g1889a165.el9 ceph 4.8 M 2024-09-17T15:43:32.388 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 17 M 2024-09-17T15:43:32.388 INFO:teuthology.orchestra.run.smithi100.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T15:43:32.388 INFO:teuthology.orchestra.run.smithi100.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T15:43:32.388 INFO:teuthology.orchestra.run.smithi100.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T15:43:32.388 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T15:43:32.388 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T15:43:32.388 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T15:43:32.388 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T15:43:32.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T15:43:32.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T15:43:32.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T15:43:32.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections 2024-09-17T15:43:32.389 INFO:teuthology.orchestra.run.smithi100.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T15:43:32.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T15:43:32.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T15:43:32.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T15:43:32.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T15:43:32.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T15:43:32.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T15:43:32.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T15:43:32.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T15:43:32.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T15:43:32.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T15:43:32.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T15:43:32.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T15:43:32.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T15:43:32.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T15:43:32.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T15:43:32.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T15:43:32.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T15:43:32.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T15:43:32.391 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T15:43:32.391 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T15:43:32.391 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:32.391 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:43:32.391 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:32.391 INFO:teuthology.orchestra.run.smithi100.stdout:Install 37 Packages 2024-09-17T15:43:32.391 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:32.393 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 31 M 2024-09-17T15:43:32.394 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 109 M 2024-09-17T15:43:32.394 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:43:33.044 INFO:teuthology.orchestra.run.smithi100.stdout:(1/37): ceph-19.3.0-4975.g1889a165.el9.x86_64.r 46 kB/s | 6.6 kB 00:00 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi100.stdout:(2/37): ceph-mgr-19.3.0-4975.g1889a165.el9.x86_ 4.7 MB/s | 1.5 MB 00:00 2024-09-17T15:43:33.317 INFO:teuthology.orchestra.run.smithi100.stdout:(3/37): ceph-mds-19.3.0-4975.g1889a165.el9.x86_ 5.7 MB/s | 2.4 MB 00:00 2024-09-17T15:43:33.368 INFO:teuthology.orchestra.run.smithi100.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4975.g1889 5.7 MB/s | 291 kB 00:00 2024-09-17T15:43:33.493 INFO:teuthology.orchestra.run.smithi100.stdout:(5/37): ceph-mon-19.3.0-4975.g1889a165.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-17T15:43:33.619 INFO:teuthology.orchestra.run.smithi100.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-17T15:43:33.677 INFO:teuthology.orchestra.run.smithi100.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-17T15:43:33.727 INFO:teuthology.orchestra.run.smithi100.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-09-17T15:43:33.778 INFO:teuthology.orchestra.run.smithi100.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-17T15:43:33.845 INFO:teuthology.orchestra.run.smithi100.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 3.5 MB/s | 1.2 MB 00:00 2024-09-17T15:43:33.870 INFO:teuthology.orchestra.run.smithi100.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-09-17T15:43:34.070 INFO:teuthology.orchestra.run.smithi100.stdout:(12/37): ceph-osd-19.3.0-4975.g1889a165.el9.x86 20 MB/s | 17 MB 00:00 2024-09-17T15:43:34.179 INFO:teuthology.orchestra.run.smithi100.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 558 kB/s | 172 kB 00:00 2024-09-17T15:43:34.204 INFO:teuthology.orchestra.run.smithi100.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 260 kB/s | 35 kB 00:00 2024-09-17T15:43:34.229 INFO:teuthology.orchestra.run.smithi100.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 490 kB/s | 188 kB 00:00 2024-09-17T15:43:34.254 INFO:teuthology.orchestra.run.smithi100.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-17T15:43:34.280 INFO:teuthology.orchestra.run.smithi100.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 674 kB/s | 51 kB 00:00 2024-09-17T15:43:34.313 INFO:teuthology.orchestra.run.smithi100.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 497 kB/s | 42 kB 00:00 2024-09-17T15:43:34.347 INFO:teuthology.orchestra.run.smithi100.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-17T15:43:34.372 INFO:teuthology.orchestra.run.smithi100.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 764 kB/s | 43 kB 00:00 2024-09-17T15:43:34.397 INFO:teuthology.orchestra.run.smithi100.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.5 MB/s | 172 kB 00:00 2024-09-17T15:43:34.423 INFO:teuthology.orchestra.run.smithi100.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 427 kB/s | 11 kB 00:00 2024-09-17T15:43:34.448 INFO:teuthology.orchestra.run.smithi100.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 713 kB/s | 18 kB 00:00 2024-09-17T15:43:34.473 INFO:teuthology.orchestra.run.smithi100.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 920 kB/s | 23 kB 00:00 2024-09-17T15:43:34.498 INFO:teuthology.orchestra.run.smithi100.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-09-17T15:43:34.524 INFO:teuthology.orchestra.run.smithi100.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-17T15:43:34.557 INFO:teuthology.orchestra.run.smithi100.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 335 kB/s | 20 kB 00:00 2024-09-17T15:43:34.583 INFO:teuthology.orchestra.run.smithi100.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 899 kB/s | 46 kB 00:00 2024-09-17T15:43:34.608 INFO:teuthology.orchestra.run.smithi100.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-17T15:43:34.633 INFO:teuthology.orchestra.run.smithi100.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 656 kB/s | 16 kB 00:00 2024-09-17T15:43:34.658 INFO:teuthology.orchestra.run.smithi100.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-17T15:43:34.684 INFO:teuthology.orchestra.run.smithi100.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-17T15:43:34.724 INFO:teuthology.orchestra.run.smithi100.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 551 kB/s | 36 kB 00:00 2024-09-17T15:43:34.751 INFO:teuthology.orchestra.run.smithi100.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-09-17T15:43:34.768 INFO:teuthology.orchestra.run.smithi100.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-17T15:43:34.801 INFO:teuthology.orchestra.run.smithi100.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.4 MB/s | 427 kB 00:00 2024-09-17T15:43:34.902 INFO:teuthology.orchestra.run.smithi100.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 34 kB/s | 22 kB 00:00 2024-09-17T15:43:34.906 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:43:34.906 INFO:teuthology.orchestra.run.smithi100.stdout:Total 12 MB/s | 31 MB 00:02 2024-09-17T15:43:35.056 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:43:35.097 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:43:35.097 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:43:35.587 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:43:35.588 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:43:36.811 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:43:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:43:36.934 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:43:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T15:43:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T15:43:37.069 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T15:43:37.140 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T15:43:37.185 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T15:43:37.217 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T15:43:37.310 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T15:43:37.405 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T15:43:37.454 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T15:43:37.489 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T15:43:37.518 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T15:43:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T15:43:37.586 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T15:43:37.627 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T15:43:37.662 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T15:43:37.696 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T15:43:37.745 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T15:43:37.777 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T15:43:37.875 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T15:43:38.014 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T15:43:38.094 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T15:43:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T15:43:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T15:43:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T15:43:38.290 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T15:43:38.476 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T15:43:38.516 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T15:43:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T15:43:38.721 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T15:43:38.732 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T15:43:38.860 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T15:43:39.021 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 32/37 2024-09-17T15:43:39.050 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:43:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:43:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 1/4 2024-09-17T15:43:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2/4 2024-09-17T15:43:39.066 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T15:43:39.077 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:43:39.077 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:43:39.077 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T15:43:39.077 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:43:39.077 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:43:39.078 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:39.387 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T15:43:39.387 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:39.388 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:43:39.388 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:39.388 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:39.388 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T15:43:39.388 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T15:43:39.388 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:39.388 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:43:39.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph 2024-09-17T15:43:40.292 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:42 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:43:40.433 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:43:40.435 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-17T15:43:40.436 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:40.436 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:43:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.3.0-4975.g1889a165.el9 ceph 6.6 k 2024-09-17T15:43:40.436 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-17T15:43:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.3.0-4975.g1889a165.el9 ceph 2.4 M 2024-09-17T15:43:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.3.0-4975.g1889a165.el9 ceph 1.5 M 2024-09-17T15:43:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 291 k 2024-09-17T15:43:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.3.0-4975.g1889a165.el9 ceph 4.8 M 2024-09-17T15:43:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 17 M 2024-09-17T15:43:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T15:43:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T15:43:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T15:43:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T15:43:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T15:43:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T15:43:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T15:43:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T15:43:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T15:43:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T15:43:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections 2024-09-17T15:43:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T15:43:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T15:43:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T15:43:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T15:43:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T15:43:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T15:43:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T15:43:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T15:43:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T15:43:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T15:43:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T15:43:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T15:43:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T15:43:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T15:43:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T15:43:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T15:43:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T15:43:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T15:43:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T15:43:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T15:43:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T15:43:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:40.439 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:43:40.439 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:40.439 INFO:teuthology.orchestra.run.smithi064.stdout:Install 37 Packages 2024-09-17T15:43:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:40.440 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 31 M 2024-09-17T15:43:40.440 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 109 M 2024-09-17T15:43:40.440 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:43:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:43:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:43:40.563 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:43:40.563 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T15:43:40.563 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:43:40.563 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:43:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:43:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:43:41.052 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:43:41.052 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T15:43:41.052 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:43:41.052 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:43:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:41.278 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:43:41.301 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:43:41.301 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:43:41.301 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T15:43:41.302 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:43:41.302 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:43:41.302 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:43:41.406 INFO:teuthology.orchestra.run.smithi064.stdout:(1/37): ceph-19.3.0-4975.g1889a165.el9.x86_64.r 54 kB/s | 6.6 kB 00:00 2024-09-17T15:43:41.581 INFO:teuthology.orchestra.run.smithi064.stdout:(2/37): ceph-mgr-19.3.0-4975.g1889a165.el9.x86_ 5.1 MB/s | 1.5 MB 00:00 2024-09-17T15:43:41.648 INFO:teuthology.orchestra.run.smithi064.stdout:(3/37): ceph-mds-19.3.0-4975.g1889a165.el9.x86_ 6.5 MB/s | 2.4 MB 00:00 2024-09-17T15:43:41.732 INFO:teuthology.orchestra.run.smithi064.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4975.g1889 3.4 MB/s | 291 kB 00:00 2024-09-17T15:43:41.816 INFO:teuthology.orchestra.run.smithi064.stdout:(5/37): ceph-mon-19.3.0-4975.g1889a165.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-17T15:43:42.166 INFO:teuthology.orchestra.run.smithi064.stdout:(6/37): ceph-osd-19.3.0-4975.g1889a165.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-17T15:43:42.384 INFO:teuthology.orchestra.run.smithi064.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 388 kB/s | 253 kB 00:00 2024-09-17T15:43:42.583 INFO:teuthology.orchestra.run.smithi064.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 681 kB/s | 135 kB 00:00 2024-09-17T15:43:42.633 INFO:teuthology.orchestra.run.smithi064.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 228 kB/s | 106 kB 00:00 2024-09-17T15:43:42.692 INFO:teuthology.orchestra.run.smithi064.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-09-17T15:43:42.726 INFO:teuthology.orchestra.run.smithi064.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-09-17T15:43:42.837 INFO:teuthology.orchestra.run.smithi064.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-09-17T15:43:42.854 INFO:teuthology.orchestra.run.smithi064.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-17T15:43:42.888 INFO:teuthology.orchestra.run.smithi064.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-17T15:43:42.905 INFO:teuthology.orchestra.run.smithi064.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-17T15:43:42.938 INFO:teuthology.orchestra.run.smithi064.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-17T15:43:42.964 INFO:teuthology.orchestra.run.smithi064.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 709 kB/s | 42 kB 00:00 2024-09-17T15:43:42.989 INFO:teuthology.orchestra.run.smithi064.stdout:(18/37): python3-markupsafe-1.1.1-12.el9.x86_64 229 kB/s | 35 kB 00:00 2024-09-17T15:43:43.039 INFO:teuthology.orchestra.run.smithi064.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-17T15:43:43.065 INFO:teuthology.orchestra.run.smithi064.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-17T15:43:43.090 INFO:teuthology.orchestra.run.smithi064.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-17T15:43:43.114 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:43:43.115 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 1/37 2024-09-17T15:43:43.115 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2/37 2024-09-17T15:43:43.115 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 3/37 2024-09-17T15:43:43.115 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 4/37 2024-09-17T15:43:43.115 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 5/37 2024-09-17T15:43:43.115 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 6/37 2024-09-17T15:43:43.115 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T15:43:43.115 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T15:43:43.115 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T15:43:43.115 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T15:43:43.115 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T15:43:43.115 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T15:43:43.115 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T15:43:43.116 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T15:43:43.116 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T15:43:43.116 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T15:43:43.116 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T15:43:43.116 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T15:43:43.116 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T15:43:43.116 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T15:43:43.116 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T15:43:43.116 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T15:43:43.116 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T15:43:43.117 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T15:43:43.117 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T15:43:43.117 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T15:43:43.117 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T15:43:43.117 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T15:43:43.117 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T15:43:43.117 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T15:43:43.117 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T15:43:43.118 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T15:43:43.118 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T15:43:43.118 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T15:43:43.118 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T15:43:43.118 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T15:43:43.124 INFO:teuthology.orchestra.run.smithi064.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 412 kB/s | 11 kB 00:00 2024-09-17T15:43:43.149 INFO:teuthology.orchestra.run.smithi064.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-09-17T15:43:43.174 INFO:teuthology.orchestra.run.smithi064.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 365 kB/s | 18 kB 00:00 2024-09-17T15:43:43.200 INFO:teuthology.orchestra.run.smithi064.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-17T15:43:43.225 INFO:teuthology.orchestra.run.smithi064.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-17T15:43:43.250 INFO:teuthology.orchestra.run.smithi064.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-17T15:43:43.275 INFO:teuthology.orchestra.run.smithi064.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-09-17T15:43:43.301 INFO:teuthology.orchestra.run.smithi064.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-17T15:43:43.326 INFO:teuthology.orchestra.run.smithi064.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 657 kB/s | 16 kB 00:00 2024-09-17T15:43:43.351 INFO:teuthology.orchestra.run.smithi064.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-09-17T15:43:43.377 INFO:teuthology.orchestra.run.smithi064.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-17T15:43:43.402 INFO:teuthology.orchestra.run.smithi064.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-09-17T15:43:43.427 INFO:teuthology.orchestra.run.smithi064.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-17T15:43:43.452 INFO:teuthology.orchestra.run.smithi064.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-09-17T15:43:43.477 INFO:teuthology.orchestra.run.smithi064.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-17T15:43:43.503 INFO:teuthology.orchestra.run.smithi064.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-17T15:43:43.507 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:43:43.507 INFO:teuthology.orchestra.run.smithi064.stdout:Total 10 MB/s | 31 MB 00:03 2024-09-17T15:43:43.662 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:43:43.706 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:43:43.706 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:43:44.065 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T15:43:44.065 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:44.065 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:43:44.065 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:44.065 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:44.065 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:44.065 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:43:44.065 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:44.065 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:44.066 INFO:teuthology.orchestra.run.smithi100.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T15:43:44.066 INFO:teuthology.orchestra.run.smithi100.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T15:43:44.066 INFO:teuthology.orchestra.run.smithi100.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T15:43:44.066 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T15:43:44.066 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T15:43:44.066 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T15:43:44.066 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T15:43:44.066 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T15:43:44.066 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T15:43:44.066 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T15:43:44.066 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T15:43:44.066 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T15:43:44.066 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T15:43:44.067 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T15:43:44.068 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T15:43:44.068 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T15:43:44.068 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T15:43:44.068 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T15:43:44.068 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:44.068 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:43:44.228 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:43:44.229 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:43:44.474 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-base 2024-09-17T15:43:45.066 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:52 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:43:45.155 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:43:45.197 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:43:45.198 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T15:43:45.198 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:43:45.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install cephadm 2024-09-17T15:43:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:43:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T15:43:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T15:43:45.836 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:52 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:43:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T15:43:45.926 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T15:43:45.967 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:43:45.967 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:45.967 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T15:43:45.967 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:45.968 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:43:45.968 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 784 k 2024-09-17T15:43:45.968 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:45.968 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:43:45.969 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:45.969 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T15:43:45.970 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:45.970 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 784 k 2024-09-17T15:43:45.970 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 791 k 2024-09-17T15:43:45.970 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:43:46.087 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T15:43:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T15:43:46.213 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T15:43:46.297 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T15:43:46.318 INFO:teuthology.orchestra.run.smithi100.stdout:cephadm-19.3.0-4975.g1889a165.el9.noarch.rpm 2.2 MB/s | 784 kB 00:00 2024-09-17T15:43:46.319 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:43:46.320 INFO:teuthology.orchestra.run.smithi100.stdout:Total 2.2 MB/s | 784 kB 00:00 2024-09-17T15:43:46.320 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:43:46.324 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:43:46.325 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:43:46.331 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:43:46.331 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:43:46.344 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T15:43:46.378 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T15:43:46.404 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:43:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T15:43:46.442 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T15:43:46.476 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T15:43:46.517 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T15:43:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T15:43:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T15:43:46.643 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T15:43:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T15:43:46.761 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T15:43:46.863 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:43:46.913 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T15:43:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:43:47.004 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T15:43:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T15:43:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T15:43:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T15:43:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T15:43:47.349 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:43:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T15:43:47.496 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T15:43:47.569 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T15:43:47.604 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:43:47.604 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:47.604 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:43:47.604 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:43:47.605 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:47.605 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:43:47.698 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T15:43:47.753 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T15:43:47.805 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T15:43:47.892 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T15:43:48.066 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 32/37 2024-09-17T15:43:48.096 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:43:48.124 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:43:48.124 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:43:48.124 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T15:43:48.124 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:43:48.124 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:43:48.124 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:55 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:43:48.528 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4975.g1889a165.el9 ceph 145 k 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T15:43:48.530 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:48.530 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 145 k 2024-09-17T15:43:48.530 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 432 k 2024-09-17T15:43:48.530 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:43:48.735 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-immutable-object-cache-19.3.0-4975.g1889a1 709 kB/s | 145 kB 00:00 2024-09-17T15:43:48.735 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:43:48.736 INFO:teuthology.orchestra.run.smithi100.stdout:Total 703 kB/s | 145 kB 00:00 2024-09-17T15:43:48.736 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:43:48.743 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:43:48.743 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:43:48.800 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:43:48.800 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:43:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:43:49.036 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:43:49.060 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:43:49.061 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:43:49.061 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T15:43:49.061 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:49.680 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:43:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:43:49.709 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:43:49.709 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T15:43:49.709 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:43:49.709 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:43:49.709 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:49.974 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:43:49.974 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:49.974 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:43:49.974 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:49.974 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:49.974 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:43:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:43:50.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr 2024-09-17T15:43:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:43:50.231 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:43:50.231 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T15:43:50.231 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:43:50.231 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:43:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:50.473 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:43:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:43:50.499 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:43:50.499 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T15:43:50.499 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:43:50.499 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:43:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:43:50.811 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:57 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:43:50.902 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:43:50.943 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:43:50.944 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T15:43:50.944 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:43:51.017 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T15:43:51.597 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:58 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:43:51.733 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:43:51.733 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:51.734 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T15:43:51.734 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:51.734 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:43:51.734 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 14 M 2024-09-17T15:43:51.734 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T15:43:51.734 INFO:teuthology.orchestra.run.smithi100.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T15:43:51.734 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 34 k 2024-09-17T15:43:51.734 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 16 k 2024-09-17T15:43:51.734 INFO:teuthology.orchestra.run.smithi100.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T15:43:51.734 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T15:43:51.734 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T15:43:51.734 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T15:43:51.734 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T15:43:51.735 INFO:teuthology.orchestra.run.smithi100.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T15:43:51.735 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T15:43:51.735 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T15:43:51.735 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:51.735 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:43:51.735 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:43:51.735 INFO:teuthology.orchestra.run.smithi100.stdout:Install 12 Packages 2024-09-17T15:43:51.735 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:43:51.736 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 19 M 2024-09-17T15:43:51.736 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 159 M 2024-09-17T15:43:51.736 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:43:52.309 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:43:52.309 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 1/37 2024-09-17T15:43:52.309 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2/37 2024-09-17T15:43:52.309 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 3/37 2024-09-17T15:43:52.310 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 4/37 2024-09-17T15:43:52.310 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 5/37 2024-09-17T15:43:52.310 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 6/37 2024-09-17T15:43:52.310 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T15:43:52.310 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T15:43:52.310 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T15:43:52.310 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T15:43:52.310 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T15:43:52.310 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T15:43:52.310 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T15:43:52.310 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T15:43:52.310 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T15:43:52.310 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T15:43:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T15:43:52.312 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T15:43:52.312 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T15:43:52.312 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T15:43:52.312 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T15:43:52.312 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T15:43:52.547 INFO:teuthology.orchestra.run.smithi100.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4975.g18 227 kB/s | 34 kB 00:00 2024-09-17T15:43:52.572 INFO:teuthology.orchestra.run.smithi100.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4975.g188 93 kB/s | 16 kB 00:00 2024-09-17T15:43:52.798 INFO:teuthology.orchestra.run.smithi100.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-17T15:43:52.864 INFO:teuthology.orchestra.run.smithi100.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-09-17T15:43:52.915 INFO:teuthology.orchestra.run.smithi100.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-17T15:43:52.940 INFO:teuthology.orchestra.run.smithi100.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 778 kB/s | 19 kB 00:00 2024-09-17T15:43:52.974 INFO:teuthology.orchestra.run.smithi100.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-09-17T15:43:53.132 INFO:teuthology.orchestra.run.smithi100.stdout:(8/12): ceph-mgr-dashboard-19.3.0-4975.g1889a16 19 MB/s | 14 MB 00:00 2024-09-17T15:43:53.166 INFO:teuthology.orchestra.run.smithi100.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 753 kB/s | 144 kB 00:00 2024-09-17T15:43:53.191 INFO:teuthology.orchestra.run.smithi100.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 544 kB/s | 31 kB 00:00 2024-09-17T15:43:53.228 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T15:43:53.228 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:53.228 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:43:53.228 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:53.228 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:53.228 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:53.228 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:43:53.228 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:53.229 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:53.229 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T15:43:53.229 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T15:43:53.229 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T15:43:53.229 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T15:43:53.229 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T15:43:53.229 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T15:43:53.229 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T15:43:53.230 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T15:43:53.230 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T15:43:53.230 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T15:43:53.230 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T15:43:53.230 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T15:43:53.230 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T15:43:53.230 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T15:43:53.230 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T15:43:53.230 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T15:43:53.230 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T15:43:53.230 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T15:43:53.230 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T15:43:53.230 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T15:43:53.231 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T15:43:53.231 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T15:43:53.231 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T15:43:53.231 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T15:43:53.231 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T15:43:53.231 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T15:43:53.231 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T15:43:53.231 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T15:43:53.231 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T15:43:53.231 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T15:43:53.232 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T15:43:53.232 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:53.232 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:43:53.241 INFO:teuthology.orchestra.run.smithi100.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-09-17T15:43:53.292 INFO:teuthology.orchestra.run.smithi100.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.8 MB/s | 2.0 MB 00:00 2024-09-17T15:43:53.296 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:43:53.296 INFO:teuthology.orchestra.run.smithi100.stdout:Total 12 MB/s | 19 MB 00:01 2024-09-17T15:43:53.347 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:43:53.366 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:43:53.366 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:43:53.586 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base 2024-09-17T15:43:53.642 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:43:53.643 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:43:54.198 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:56 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:43:54.244 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:43:54.291 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:43:54.326 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-17T15:43:54.336 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:43:54.337 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-17T15:43:54.337 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:43:54.369 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-17T15:43:54.416 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-17T15:43:54.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install cephadm 2024-09-17T15:43:54.454 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-17T15:43:54.568 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-17T15:43:54.697 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-17T15:43:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-17T15:43:54.827 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-17T15:43:54.854 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:43:54.887 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 10/12 2024-09-17T15:43:55.023 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:57 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:43:55.159 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:43:55.160 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:55.160 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-17T15:43:55.160 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:55.160 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:43:55.160 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 784 k 2024-09-17T15:43:55.160 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:55.160 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:43:55.160 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:55.161 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-17T15:43:55.161 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:55.161 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 784 k 2024-09-17T15:43:55.161 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 791 k 2024-09-17T15:43:55.161 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:43:55.430 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm-19.3.0-4975.g1889a165.el9.noarch.rpm 2.9 MB/s | 784 kB 00:00 2024-09-17T15:43:55.430 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:43:55.430 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.8 MB/s | 784 kB 00:00 2024-09-17T15:43:55.431 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:43:55.436 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:43:55.436 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:43:55.442 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:43:55.443 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:43:55.517 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:43:55.936 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:43:56.028 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:43:56.410 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:43:56.626 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:43:56.627 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:56.627 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:43:56.627 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:43:56.627 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:56.627 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:43:56.825 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T15:43:57.434 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:43:57.575 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:43:57.576 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-17T15:43:57.576 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:57.576 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:43:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4975.g1889a165.el9 ceph 145 k 2024-09-17T15:43:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:57.577 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:43:57.577 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:43:57.577 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-17T15:43:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:57.577 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 145 k 2024-09-17T15:43:57.577 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 432 k 2024-09-17T15:43:57.577 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:43:57.943 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-19.3.0-4975.g1889a1 395 kB/s | 145 kB 00:00 2024-09-17T15:43:57.944 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:43:57.944 INFO:teuthology.orchestra.run.smithi064.stdout:Total 394 kB/s | 145 kB 00:00 2024-09-17T15:43:57.944 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:43:57.952 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:43:57.953 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:43:58.013 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:43:58.013 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:43:58.060 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 11/12 2024-09-17T15:43:58.116 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:43:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:43:58.228 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:43:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:43:58.254 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:43:58.254 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T15:43:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:59.136 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:43:59.136 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:59.136 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:43:59.136 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:43:59.136 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:43:59.136 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:43:59.367 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr 2024-09-17T15:43:59.981 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:01 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:44:00.072 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:44:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:44:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 1/12 2024-09-17T15:44:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 2/12 2024-09-17T15:44:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 3/12 2024-09-17T15:44:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-17T15:44:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-17T15:44:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-17T15:44:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-17T15:44:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-17T15:44:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:44:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-17T15:44:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-17T15:44:00.116 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:44:00.117 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-17T15:44:00.117 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:44:00.197 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T15:44:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-17T15:44:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:00.431 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:44:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T15:44:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T15:44:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T15:44:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T15:44:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T15:44:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T15:44:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T15:44:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T15:44:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T15:44:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:00.432 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:00.697 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T15:44:00.799 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:02 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:44:00.939 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:44:00.941 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-17T15:44:00.941 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:00.941 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:44:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 14 M 2024-09-17T15:44:00.941 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-17T15:44:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T15:44:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 34 k 2024-09-17T15:44:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 16 k 2024-09-17T15:44:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T15:44:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T15:44:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T15:44:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T15:44:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T15:44:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T15:44:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T15:44:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T15:44:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:00.943 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:44:00.943 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:00.943 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-09-17T15:44:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:00.944 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 19 M 2024-09-17T15:44:00.944 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 159 M 2024-09-17T15:44:00.944 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:44:01.283 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:08 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:01.425 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:01.426 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-09-17T15:44:01.427 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T15:44:01.427 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-09-17T15:44:01.427 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:44:01.427 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 7.4 M 2024-09-17T15:44:01.427 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T15:44:01.427 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T15:44:01.427 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T15:44:01.427 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T15:44:01.427 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T15:44:01.427 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T15:44:01.427 INFO:teuthology.orchestra.run.smithi100.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T15:44:01.427 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T15:44:01.428 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T15:44:01.428 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T15:44:01.428 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T15:44:01.428 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T15:44:01.428 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:01.428 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:44:01.428 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-09-17T15:44:01.428 INFO:teuthology.orchestra.run.smithi100.stdout:Install 12 Packages 2024-09-17T15:44:01.428 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:01.429 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 38 M 2024-09-17T15:44:01.429 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 212 M 2024-09-17T15:44:01.429 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:44:01.647 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4975.g18 237 kB/s | 34 kB 00:00 2024-09-17T15:44:01.672 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4975.g188 97 kB/s | 16 kB 00:00 2024-09-17T15:44:01.839 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 269 kB 00:00 2024-09-17T15:44:01.898 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-09-17T15:44:01.948 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-17T15:44:01.949 INFO:teuthology.orchestra.run.smithi100.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.6 MB/s | 191 kB 00:00 2024-09-17T15:44:01.973 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 786 kB/s | 19 kB 00:00 2024-09-17T15:44:01.982 INFO:teuthology.orchestra.run.smithi100.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 986 kB/s | 33 kB 00:00 2024-09-17T15:44:02.016 INFO:teuthology.orchestra.run.smithi100.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.2 MB/s | 801 kB 00:00 2024-09-17T15:44:02.023 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.6 MB/s | 863 kB 00:00 2024-09-17T15:44:02.049 INFO:teuthology.orchestra.run.smithi100.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 546 kB/s | 18 kB 00:00 2024-09-17T15:44:02.082 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 19 MB/s | 2.0 MB 00:00 2024-09-17T15:44:02.083 INFO:teuthology.orchestra.run.smithi100.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 1.1 MB/s | 38 kB 00:00 2024-09-17T15:44:02.107 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.3 MB/s | 31 kB 00:00 2024-09-17T15:44:02.133 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.3 MB/s | 144 kB 00:00 2024-09-17T15:44:02.133 INFO:teuthology.orchestra.run.smithi100.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 20 MB/s | 3.0 MB 00:00 2024-09-17T15:44:02.175 INFO:teuthology.orchestra.run.smithi100.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 5.7 MB/s | 245 kB 00:00 2024-09-17T15:44:02.277 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): ceph-mgr-dashboard-19.3.0-4975.g1889a1 18 MB/s | 14 MB 00:00 2024-09-17T15:44:02.302 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 970 kB/s | 188 kB 00:00 2024-09-17T15:44:02.305 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:02.306 INFO:teuthology.orchestra.run.smithi064.stdout:Total 14 MB/s | 19 MB 00:01 2024-09-17T15:44:02.326 INFO:teuthology.orchestra.run.smithi100.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 5.1 MB 00:00 2024-09-17T15:44:02.356 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:44:02.376 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:44:02.376 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:44:02.401 INFO:teuthology.orchestra.run.smithi100.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 16 MB/s | 4.9 MB 00:00 2024-09-17T15:44:02.426 INFO:teuthology.orchestra.run.smithi100.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-09-17T15:44:02.543 INFO:teuthology.orchestra.run.smithi100.stdout:(11/12): ceph-mgr-diskprediction-local-19.3.0-4 10 MB/s | 7.4 MB 00:00 2024-09-17T15:44:02.666 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:44:02.666 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:44:02.885 INFO:teuthology.orchestra.run.smithi100.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-09-17T15:44:02.886 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:02.886 INFO:teuthology.orchestra.run.smithi100.stdout:Total 26 MB/s | 38 MB 00:01 2024-09-17T15:44:03.240 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:44:03.268 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:44:03.268 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:44:03.275 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:44:03.361 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-17T15:44:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-17T15:44:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-17T15:44:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-17T15:44:03.589 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-17T15:44:03.699 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:44:03.699 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:44:03.722 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-17T15:44:03.814 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-17T15:44:03.875 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-17T15:44:04.056 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:44:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 10/12 2024-09-17T15:44:04.457 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:44:04.540 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T15:44:04.583 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:44:04.683 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:44:04.726 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T15:44:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T15:44:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T15:44:05.035 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T15:44:05.678 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T15:44:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T15:44:07.037 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:44:07.487 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 11/12 2024-09-17T15:44:07.559 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:44:08.073 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T15:44:08.109 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:44:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:44:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 1/12 2024-09-17T15:44:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 2/12 2024-09-17T15:44:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 3/12 2024-09-17T15:44:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-17T15:44:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-17T15:44:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-17T15:44:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-17T15:44:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-17T15:44:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:44:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-17T15:44:09.263 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-17T15:44:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-17T15:44:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:09.593 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:09.594 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:44:09.864 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T15:44:10.496 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:12 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:44:10.633 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:44:10.634 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-09-17T15:44:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-17T15:44:10.635 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-09-17T15:44:10.635 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:44:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 7.4 M 2024-09-17T15:44:10.635 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-17T15:44:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T15:44:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T15:44:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T15:44:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T15:44:10.635 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T15:44:10.636 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T15:44:10.636 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T15:44:10.636 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T15:44:10.636 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T15:44:10.636 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T15:44:10.636 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T15:44:10.636 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:10.636 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:44:10.636 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-09-17T15:44:10.636 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-09-17T15:44:10.636 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:10.637 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 38 M 2024-09-17T15:44:10.637 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 212 M 2024-09-17T15:44:10.637 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:44:10.664 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:44:10.664 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 1/12 2024-09-17T15:44:10.664 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:44:10.664 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:44:10.664 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T15:44:10.664 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T15:44:10.665 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T15:44:10.665 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T15:44:10.665 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T15:44:10.665 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T15:44:10.665 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:44:10.665 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T15:44:11.125 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T15:44:11.125 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:11.126 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:44:11.126 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:11.126 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T15:44:11.126 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T15:44:11.126 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T15:44:11.126 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T15:44:11.126 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T15:44:11.126 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T15:44:11.126 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T15:44:11.126 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T15:44:11.126 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T15:44:11.126 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T15:44:11.126 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T15:44:11.127 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:11.127 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:11.417 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-17T15:44:11.509 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 349 kB/s | 191 kB 00:00 2024-09-17T15:44:11.534 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-rook 2024-09-17T15:44:11.710 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 112 kB/s | 33 kB 00:00 2024-09-17T15:44:11.777 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): libgfortran-11.5.0-2.el9.x86_64.rpm 983 kB/s | 801 kB 00:00 2024-09-17T15:44:11.819 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 167 kB/s | 18 kB 00:00 2024-09-17T15:44:11.885 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 354 kB/s | 38 kB 00:00 2024-09-17T15:44:12.011 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-17T15:44:12.136 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:19 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:12.186 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.4 MB/s | 3.0 MB 00:00 2024-09-17T15:44:12.268 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:12.269 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T15:44:12.270 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:12.270 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:44:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 49 k 2024-09-17T15:44:12.270 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T15:44:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T15:44:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T15:44:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T15:44:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T15:44:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T15:44:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T15:44:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T15:44:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T15:44:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T15:44:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T15:44:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T15:44:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T15:44:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:12.271 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:44:12.271 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:12.271 INFO:teuthology.orchestra.run.smithi100.stdout:Install 13 Packages 2024-09-17T15:44:12.272 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:12.272 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 2.2 M 2024-09-17T15:44:12.272 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 26 M 2024-09-17T15:44:12.272 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:44:12.303 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-09-17T15:44:12.592 INFO:teuthology.orchestra.run.smithi100.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 232 kB/s | 26 kB 00:00 2024-09-17T15:44:12.603 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.6 MB/s | 5.1 MB 00:00 2024-09-17T15:44:12.617 INFO:teuthology.orchestra.run.smithi100.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 141 kB/s | 19 kB 00:00 2024-09-17T15:44:12.701 INFO:teuthology.orchestra.run.smithi100.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-09-17T15:44:12.726 INFO:teuthology.orchestra.run.smithi100.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-17T15:44:12.768 INFO:teuthology.orchestra.run.smithi100.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 279 kB 00:00 2024-09-17T15:44:12.793 INFO:teuthology.orchestra.run.smithi100.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 801 kB/s | 54 kB 00:00 2024-09-17T15:44:12.819 INFO:teuthology.orchestra.run.smithi100.stdout:(7/13): ceph-mgr-rook-19.3.0-4975.g1889a165.el9 146 kB/s | 49 kB 00:00 2024-09-17T15:44:12.869 INFO:teuthology.orchestra.run.smithi100.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-17T15:44:12.894 INFO:teuthology.orchestra.run.smithi100.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-17T15:44:12.919 INFO:teuthology.orchestra.run.smithi100.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-17T15:44:12.937 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.4 MB/s | 4.9 MB 00:01 2024-09-17T15:44:12.945 INFO:teuthology.orchestra.run.smithi100.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-17T15:44:12.978 INFO:teuthology.orchestra.run.smithi100.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-17T15:44:13.020 INFO:teuthology.orchestra.run.smithi100.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-17T15:44:13.024 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:13.024 INFO:teuthology.orchestra.run.smithi100.stdout:Total 2.9 MB/s | 2.2 MB 00:00 2024-09-17T15:44:13.086 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:44:13.101 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:44:13.101 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:44:13.189 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:44:13.190 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:44:13.701 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:44:13.863 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T15:44:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T15:44:13.970 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T15:44:14.006 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T15:44:14.048 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T15:44:14.158 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T15:44:14.259 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T15:44:14.315 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T15:44:15.045 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T15:44:15.096 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T15:44:15.130 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T15:44:15.178 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T15:44:15.208 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:44:15.950 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:44:15.950 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 1/13 2024-09-17T15:44:15.950 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T15:44:15.950 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T15:44:15.951 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T15:44:15.951 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T15:44:15.951 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T15:44:15.951 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T15:44:15.951 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T15:44:15.951 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T15:44:15.951 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T15:44:15.951 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T15:44:15.951 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T15:44:16.278 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T15:44:16.278 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:16.278 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:44:16.278 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:16.278 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T15:44:16.278 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T15:44:16.278 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T15:44:16.278 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T15:44:16.279 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T15:44:16.279 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T15:44:16.279 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T15:44:16.279 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T15:44:16.279 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T15:44:16.279 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T15:44:16.279 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T15:44:16.279 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T15:44:16.279 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:16.279 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:16.526 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T15:44:17.103 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:24 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:17.235 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:17.235 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T15:44:17.236 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:17.236 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:44:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 154 k 2024-09-17T15:44:17.236 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T15:44:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T15:44:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T15:44:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T15:44:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T15:44:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T15:44:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:17.236 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:44:17.236 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:17.236 INFO:teuthology.orchestra.run.smithi100.stdout:Install 6 Packages 2024-09-17T15:44:17.237 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:17.237 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 7.0 M 2024-09-17T15:44:17.237 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 33 M 2024-09-17T15:44:17.237 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:44:17.797 INFO:teuthology.orchestra.run.smithi100.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4975.g1889a165.e 721 kB/s | 154 kB 00:00 2024-09-17T15:44:17.864 INFO:teuthology.orchestra.run.smithi100.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.0 MB/s | 548 kB 00:00 2024-09-17T15:44:17.889 INFO:teuthology.orchestra.run.smithi100.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-17T15:44:17.938 INFO:teuthology.orchestra.run.smithi100.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-17T15:44:18.357 INFO:teuthology.orchestra.run.smithi100.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 321 kB/s | 248 kB 00:00 2024-09-17T15:44:18.740 INFO:teuthology.orchestra.run.smithi100.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.2 MB/s | 6.0 MB 00:01 2024-09-17T15:44:18.742 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:18.743 INFO:teuthology.orchestra.run.smithi100.stdout:Total 4.7 MB/s | 7.0 MB 00:01 2024-09-17T15:44:18.817 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:44:18.829 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:44:18.829 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:44:18.930 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:44:18.931 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:44:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:44:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T15:44:19.458 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T15:44:19.981 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T15:44:20.069 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T15:44:20.138 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T15:44:20.174 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:44:20.945 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:44:20.945 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/6 2024-09-17T15:44:20.945 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T15:44:20.945 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T15:44:20.945 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T15:44:20.945 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T15:44:21.243 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T15:44:21.243 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:21.243 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:44:21.243 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:21.243 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T15:44:21.243 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T15:44:21.243 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T15:44:21.243 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T15:44:21.243 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T15:44:21.244 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:21.244 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:21.471 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-fuse 2024-09-17T15:44:22.044 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:29 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:22.175 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:22.175 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:22.176 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T15:44:22.176 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:22.176 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:44:22.176 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 899 k 2024-09-17T15:44:22.176 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T15:44:22.176 INFO:teuthology.orchestra.run.smithi100.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T15:44:22.176 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:22.176 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:44:22.176 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:22.176 INFO:teuthology.orchestra.run.smithi100.stdout:Install 2 Packages 2024-09-17T15:44:22.176 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:22.177 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 979 k 2024-09-17T15:44:22.177 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 2.9 M 2024-09-17T15:44:22.177 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:44:22.500 INFO:teuthology.orchestra.run.smithi100.stdout:(1/2): ceph-fuse-19.3.0-4975.g1889a165.el9.x86_ 3.3 MB/s | 899 kB 00:00 2024-09-17T15:44:22.683 INFO:teuthology.orchestra.run.smithi100.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 176 kB/s | 80 kB 00:00 2024-09-17T15:44:22.684 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:22.684 INFO:teuthology.orchestra.run.smithi100.stdout:Total 1.9 MB/s | 979 kB 00:00 2024-09-17T15:44:22.695 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:44:22.716 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:44:22.717 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:44:22.798 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:44:22.798 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:44:22.986 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:44:23.119 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T15:44:23.190 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:44:23.878 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:44:23.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/2 2024-09-17T15:44:24.170 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T15:44:24.170 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:24.170 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:44:24.170 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T15:44:24.170 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:24.170 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:24.446 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-volume 2024-09-17T15:44:25.041 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:32 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:25.177 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 285 k 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 285 k 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 1.3 M 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:44:25.388 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-volume-19.3.0-4975.g1889a165.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-17T15:44:25.389 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:25.390 INFO:teuthology.orchestra.run.smithi100.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-17T15:44:25.390 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:44:25.397 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:44:25.397 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:44:25.429 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:44:25.429 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:44:25.646 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:44:25.707 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:44:25.723 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:44:25.723 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:44:25.723 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T15:44:25.723 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:44:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:26.732 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:44:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:26.732 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:27.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados-devel 2024-09-17T15:44:27.614 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:34 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:27.748 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:27.748 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:27.749 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-17T15:44:27.749 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:27.749 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:44:27.749 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 126 k 2024-09-17T15:44:27.749 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:27.749 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:44:27.749 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:27.749 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T15:44:27.749 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:27.749 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 126 k 2024-09-17T15:44:27.749 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 456 k 2024-09-17T15:44:27.749 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:44:27.960 INFO:teuthology.orchestra.run.smithi100.stdout:librados-devel-19.3.0-4975.g1889a165.el9.x86_64 603 kB/s | 126 kB 00:00 2024-09-17T15:44:27.961 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:27.961 INFO:teuthology.orchestra.run.smithi100.stdout:Total 597 kB/s | 126 kB 00:00 2024-09-17T15:44:27.961 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:44:27.966 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:44:27.966 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:44:28.031 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:44:28.031 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:44:28.185 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:44:28.291 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:28.700 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:28.951 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:28.952 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:28.952 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:44:28.952 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:44:28.952 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:28.952 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:29.212 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs2 2024-09-17T15:44:29.802 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:36 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:29.893 INFO:teuthology.orchestra.run.smithi100.stdout:Package libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:44:29.938 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:29.940 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T15:44:29.940 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:30.008 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs-devel 2024-09-17T15:44:30.582 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:37 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:30.713 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:30.714 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:30.715 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-17T15:44:30.715 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:30.715 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:44:30.715 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 32 k 2024-09-17T15:44:30.715 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:30.716 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:44:30.716 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:30.716 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T15:44:30.716 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:30.716 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 32 k 2024-09-17T15:44:30.716 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 142 k 2024-09-17T15:44:30.717 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:44:30.768 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 901 kB/s | 16 MB 00:18 2024-09-17T15:44:30.769 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:30.770 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.9 MB/s | 38 MB 00:20 2024-09-17T15:44:30.872 INFO:teuthology.orchestra.run.smithi100.stdout:libcephfs-devel-19.3.0-4975.g1889a165.el9.x86_6 202 kB/s | 32 kB 00:00 2024-09-17T15:44:30.873 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:30.873 INFO:teuthology.orchestra.run.smithi100.stdout:Total 200 kB/s | 32 kB 00:00 2024-09-17T15:44:30.873 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:44:30.876 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:44:30.876 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:44:30.898 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:44:30.899 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:44:30.991 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:44:31.101 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:31.140 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:44:31.167 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:44:31.167 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:44:31.447 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:31.633 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:44:31.633 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:44:31.674 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:31.674 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:31.674 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:44:31.674 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:44:31.674 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:31.674 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:31.859 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados2 2024-09-17T15:44:32.426 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:44:32.467 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:32.502 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T15:44:32.550 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:44:32.558 INFO:teuthology.orchestra.run.smithi100.stdout:Package librados2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:44:32.603 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:32.604 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T15:44:32.604 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:32.654 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:44:32.676 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librbd1 2024-09-17T15:44:32.693 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T15:44:32.884 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T15:44:32.916 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T15:44:33.002 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T15:44:33.270 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:40 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:33.358 INFO:teuthology.orchestra.run.smithi100.stdout:Package librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:44:33.402 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:33.403 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T15:44:33.403 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:33.476 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rados 2024-09-17T15:44:33.695 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T15:44:33.826 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T15:44:34.053 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:34.141 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:44:34.185 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:34.186 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T15:44:34.186 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:34.259 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rgw 2024-09-17T15:44:34.842 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:34.933 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:44:34.975 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:34.976 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T15:44:34.976 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:35.053 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-cephfs 2024-09-17T15:44:35.144 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:44:35.621 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:35.709 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:44:35.752 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:35.753 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T15:44:35.753 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:35.825 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rbd 2024-09-17T15:44:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T15:44:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:44:36.397 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:36.485 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:44:36.528 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:36.528 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T15:44:36.529 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:36.597 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-fuse 2024-09-17T15:44:37.176 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:37.307 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:37.308 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:37.308 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-17T15:44:37.308 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:37.308 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:44:37.308 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 85 k 2024-09-17T15:44:37.308 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:37.309 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:44:37.309 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:37.309 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T15:44:37.309 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:37.309 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 85 k 2024-09-17T15:44:37.309 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 231 k 2024-09-17T15:44:37.309 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:44:37.492 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-fuse-19.3.0-4975.g1889a165.el9.x86_64.rpm 466 kB/s | 85 kB 00:00 2024-09-17T15:44:37.492 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:37.493 INFO:teuthology.orchestra.run.smithi100.stdout:Total 463 kB/s | 85 kB 00:00 2024-09-17T15:44:37.493 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:44:37.499 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:44:37.499 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:44:37.563 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:44:37.563 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:44:37.705 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:44:37.806 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:38.216 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:38.468 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:38.468 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:38.468 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:44:38.468 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:44:38.469 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:38.469 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:38.745 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-mirror 2024-09-17T15:44:38.785 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:44:38.785 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 1/12 2024-09-17T15:44:38.785 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:44:38.785 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:44:38.785 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T15:44:38.785 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T15:44:38.785 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T15:44:38.785 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T15:44:38.785 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T15:44:38.785 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T15:44:38.785 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:44:38.786 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T15:44:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T15:44:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:39.218 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:44:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T15:44:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T15:44:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T15:44:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T15:44:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T15:44:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T15:44:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T15:44:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T15:44:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T15:44:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T15:44:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T15:44:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:39.220 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:44:39.326 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:39.465 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:39.465 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:39.466 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-17T15:44:39.466 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:39.466 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:44:39.466 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.2 M 2024-09-17T15:44:39.466 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:39.466 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:44:39.466 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:39.466 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T15:44:39.466 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:39.467 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 3.2 M 2024-09-17T15:44:39.467 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 14 M 2024-09-17T15:44:39.467 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:44:39.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-rook 2024-09-17T15:44:39.804 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-mirror-19.3.0-4975.g1889a165.el9.x86_64.rpm 9.5 MB/s | 3.2 MB 00:00 2024-09-17T15:44:39.804 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:39.805 INFO:teuthology.orchestra.run.smithi100.stdout:Total 9.4 MB/s | 3.2 MB 00:00 2024-09-17T15:44:39.805 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:44:39.814 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:44:39.814 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:44:39.899 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:44:39.899 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:44:40.224 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:44:40.345 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:44:40.367 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:44:40.368 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:40.368 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-17T15:44:40.368 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:40.368 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:44:40.368 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 49 k 2024-09-17T15:44:40.369 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-17T15:44:40.369 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T15:44:40.369 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T15:44:40.369 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T15:44:40.369 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T15:44:40.369 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T15:44:40.369 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T15:44:40.369 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T15:44:40.369 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T15:44:40.369 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T15:44:40.369 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T15:44:40.369 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T15:44:40.369 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T15:44:40.370 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:40.370 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:44:40.370 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:40.370 INFO:teuthology.orchestra.run.smithi064.stdout:Install 13 Packages 2024-09-17T15:44:40.370 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:40.370 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 2.2 M 2024-09-17T15:44:40.370 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-09-17T15:44:40.371 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:44:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:40.439 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:44:40.439 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T15:44:40.439 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:44:40.439 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:44:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:40.567 INFO:teuthology.orchestra.run.smithi064.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 286 kB/s | 26 kB 00:00 2024-09-17T15:44:40.592 INFO:teuthology.orchestra.run.smithi064.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 167 kB/s | 19 kB 00:00 2024-09-17T15:44:40.684 INFO:teuthology.orchestra.run.smithi064.stdout:(3/13): ceph-mgr-rook-19.3.0-4975.g1889a165.el9 238 kB/s | 49 kB 00:00 2024-09-17T15:44:40.726 INFO:teuthology.orchestra.run.smithi064.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-17T15:44:40.751 INFO:teuthology.orchestra.run.smithi064.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-17T15:44:40.784 INFO:teuthology.orchestra.run.smithi064.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 918 kB/s | 54 kB 00:00 2024-09-17T15:44:40.826 INFO:teuthology.orchestra.run.smithi064.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 428 kB/s | 32 kB 00:00 2024-09-17T15:44:40.852 INFO:teuthology.orchestra.run.smithi064.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 224 kB/s | 14 kB 00:00 2024-09-17T15:44:40.877 INFO:teuthology.orchestra.run.smithi064.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.6 MB/s | 220 kB 00:00 2024-09-17T15:44:40.902 INFO:teuthology.orchestra.run.smithi064.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-17T15:44:40.944 INFO:teuthology.orchestra.run.smithi064.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-09-17T15:44:40.978 INFO:teuthology.orchestra.run.smithi064.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-09-17T15:44:41.011 INFO:teuthology.orchestra.run.smithi064.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-09-17T15:44:41.014 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:41.014 INFO:teuthology.orchestra.run.smithi064.stdout:Total 3.4 MB/s | 2.2 MB 00:00 2024-09-17T15:44:41.082 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:44:41.098 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:44:41.098 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:44:41.194 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:44:41.196 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:44:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:41.484 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:44:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:44:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:41.484 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:41.711 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:44:41.751 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-nbd 2024-09-17T15:44:41.899 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T15:44:41.955 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T15:44:42.012 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T15:44:42.055 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T15:44:42.097 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T15:44:42.203 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T15:44:42.314 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T15:44:42.349 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:49 ago on Tue 17 Sep 2024 03:42:53 PM UTC. 2024-09-17T15:44:42.416 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T15:44:42.482 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T15:44:42.483 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:42.483 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-17T15:44:42.483 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:42.483 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T15:44:42.483 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 170 k 2024-09-17T15:44:42.484 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:42.484 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T15:44:42.484 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T15:44:42.484 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T15:44:42.484 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:42.484 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 170 k 2024-09-17T15:44:42.484 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 490 k 2024-09-17T15:44:42.484 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T15:44:42.682 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-nbd-19.3.0-4975.g1889a165.el9.x86_64.rpm 859 kB/s | 170 kB 00:00 2024-09-17T15:44:42.683 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:42.683 INFO:teuthology.orchestra.run.smithi100.stdout:Total 852 kB/s | 170 kB 00:00 2024-09-17T15:44:42.683 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T15:44:42.691 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T15:44:42.691 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T15:44:42.755 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T15:44:42.755 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T15:44:42.903 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T15:44:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:43.171 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T15:44:43.221 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T15:44:43.254 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T15:44:43.303 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T15:44:43.332 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:44:43.415 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:43.675 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:43.675 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:43.675 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T15:44:43.675 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:44:43.675 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:44:43.675 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T15:44:43.920 DEBUG:teuthology.parallel:result is None 2024-09-17T15:44:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:44:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 1/13 2024-09-17T15:44:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T15:44:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T15:44:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T15:44:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T15:44:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T15:44:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T15:44:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T15:44:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T15:44:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T15:44:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T15:44:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T15:44:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T15:44:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:44.347 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:44:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T15:44:44.347 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T15:44:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T15:44:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T15:44:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T15:44:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T15:44:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T15:44:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T15:44:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T15:44:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T15:44:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T15:44:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T15:44:44.348 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:44.348 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:44:44.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T15:44:45.213 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:47 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:44:45.352 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:44:45.353 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:45.353 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-17T15:44:45.353 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:45.353 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:44:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 154 k 2024-09-17T15:44:45.354 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-17T15:44:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T15:44:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T15:44:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T15:44:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T15:44:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T15:44:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:45.354 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:44:45.354 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:45.354 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-09-17T15:44:45.354 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:45.355 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 7.0 M 2024-09-17T15:44:45.355 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 33 M 2024-09-17T15:44:45.355 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:44:45.971 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-09-17T15:44:46.030 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-17T15:44:46.055 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-17T15:44:46.080 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): ceph-mgr-cephadm-19.3.0-4975.g1889a165.e 637 kB/s | 154 kB 00:00 2024-09-17T15:44:46.105 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-17T15:44:46.231 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-17T15:44:46.233 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:46.233 INFO:teuthology.orchestra.run.smithi064.stdout:Total 8.0 MB/s | 7.0 MB 00:00 2024-09-17T15:44:46.306 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:44:46.318 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:44:46.318 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:44:46.424 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:44:46.425 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:44:46.735 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:44:46.921 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T15:44:46.976 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T15:44:47.529 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T15:44:47.623 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T15:44:47.739 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T15:44:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:44:48.539 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:44:48.539 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/6 2024-09-17T15:44:48.539 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T15:44:48.539 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T15:44:48.540 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T15:44:48.540 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T15:44:48.833 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T15:44:48.833 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:48.833 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:44:48.833 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:48.833 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T15:44:48.833 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T15:44:48.833 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T15:44:48.833 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T15:44:48.834 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T15:44:48.834 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:48.834 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:44:49.092 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse 2024-09-17T15:44:49.700 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:51 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:44:49.839 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:44:49.840 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:49.840 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-17T15:44:49.840 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:49.840 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:44:49.841 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 899 k 2024-09-17T15:44:49.841 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-17T15:44:49.841 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T15:44:49.841 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:49.841 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:44:49.841 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:49.841 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-09-17T15:44:49.841 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:49.842 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 979 k 2024-09-17T15:44:49.842 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.9 M 2024-09-17T15:44:49.842 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:44:50.016 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 663 kB/s | 80 kB 00:00 2024-09-17T15:44:50.158 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-fuse-19.3.0-4975.g1889a165.el9.x86_ 3.3 MB/s | 899 kB 00:00 2024-09-17T15:44:50.158 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:50.159 INFO:teuthology.orchestra.run.smithi064.stdout:Total 3.0 MB/s | 979 kB 00:00 2024-09-17T15:44:50.171 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:44:50.194 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:44:50.194 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:44:50.281 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:44:50.282 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:44:50.476 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:44:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T15:44:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:44:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:44:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/2 2024-09-17T15:44:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T15:44:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:51.657 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:44:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T15:44:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:51.657 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:44:51.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-volume 2024-09-17T15:44:52.543 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:54 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:44:52.682 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:44:52.683 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:52.683 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-17T15:44:52.683 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:52.683 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:44:52.683 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 285 k 2024-09-17T15:44:52.683 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:52.683 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:44:52.683 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:52.683 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-17T15:44:52.683 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:52.683 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 285 k 2024-09-17T15:44:52.684 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.3 M 2024-09-17T15:44:52.684 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:44:52.896 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-volume-19.3.0-4975.g1889a165.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-17T15:44:52.896 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:52.897 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-17T15:44:52.897 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:44:52.906 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:44:52.906 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:44:52.940 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:44:52.940 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:44:53.174 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:44:53.247 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:44:53.265 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:44:53.265 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:44:53.265 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T15:44:53.265 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:54.283 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:44:54.283 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:54.283 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:44:54.283 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:44:54.283 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:54.283 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:44:54.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel 2024-09-17T15:44:55.228 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:57 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:44:55.366 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:44:55.366 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:55.366 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-17T15:44:55.366 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:55.367 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:44:55.367 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 126 k 2024-09-17T15:44:55.367 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:55.367 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:44:55.367 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:55.367 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-17T15:44:55.367 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:55.367 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 126 k 2024-09-17T15:44:55.367 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 456 k 2024-09-17T15:44:55.367 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:44:55.584 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-19.3.0-4975.g1889a165.el9.x86_64 586 kB/s | 126 kB 00:00 2024-09-17T15:44:55.584 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:55.584 INFO:teuthology.orchestra.run.smithi064.stdout:Total 581 kB/s | 126 kB 00:00 2024-09-17T15:44:55.585 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:44:55.590 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:44:55.591 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:44:55.658 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:44:55.658 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:44:55.822 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:44:55.926 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:56.360 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:56.585 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:56.586 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:56.586 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:44:56.586 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:44:56.586 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:56.586 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:44:56.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2 2024-09-17T15:44:57.465 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:59 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:44:57.558 INFO:teuthology.orchestra.run.smithi064.stdout:Package libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:44:57.604 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:44:57.608 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-17T15:44:57.609 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:44:57.688 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs-devel 2024-09-17T15:44:58.289 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:00 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:44:58.428 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:44:58.428 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:58.429 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-17T15:44:58.429 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:58.429 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:44:58.429 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 32 k 2024-09-17T15:44:58.429 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:58.429 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:44:58.429 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:44:58.429 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-17T15:44:58.429 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:58.430 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 32 k 2024-09-17T15:44:58.430 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 142 k 2024-09-17T15:44:58.430 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:44:58.587 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs-devel-19.3.0-4975.g1889a165.el9.x86_6 202 kB/s | 32 kB 00:00 2024-09-17T15:44:58.587 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:44:58.588 INFO:teuthology.orchestra.run.smithi064.stdout:Total 200 kB/s | 32 kB 00:00 2024-09-17T15:44:58.588 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:44:58.591 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:44:58.591 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:44:58.615 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:44:58.615 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:44:58.710 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:44:58.812 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:59.412 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:44:59.412 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:59.412 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:44:59.412 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:44:59.412 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:44:59.412 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:44:59.626 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2 2024-09-17T15:45:00.231 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:02 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:45:00.324 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:45:00.371 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:45:00.372 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-17T15:45:00.372 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:45:00.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1 2024-09-17T15:45:01.052 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:03 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:45:01.143 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:45:01.190 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:45:01.191 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-17T15:45:01.191 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:45:01.273 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados 2024-09-17T15:45:01.877 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:03 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:45:01.970 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:45:02.016 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:45:02.017 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-17T15:45:02.017 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:45:02.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw 2024-09-17T15:45:02.712 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:04 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:45:02.804 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:45:02.851 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:45:02.852 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-17T15:45:02.853 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:45:02.940 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs 2024-09-17T15:45:03.540 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:05 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:45:03.633 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:45:03.680 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:45:03.681 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-17T15:45:03.681 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:45:03.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd 2024-09-17T15:45:04.361 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:06 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:45:04.455 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:45:04.502 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:45:04.503 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-17T15:45:04.503 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:45:04.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:07 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:45:05.330 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:45:05.331 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:45:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-09-17T15:45:05.331 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:45:05.331 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:45:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 85 k 2024-09-17T15:45:05.331 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:05.331 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:45:05.332 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:45:05.332 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-17T15:45:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:05.332 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 85 k 2024-09-17T15:45:05.332 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 231 k 2024-09-17T15:45:05.332 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:45:05.506 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-19.3.0-4975.g1889a165.el9.x86_64.rpm 489 kB/s | 85 kB 00:00 2024-09-17T15:45:05.507 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:45:05.507 INFO:teuthology.orchestra.run.smithi064.stdout:Total 485 kB/s | 85 kB 00:00 2024-09-17T15:45:05.507 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:45:05.514 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:45:05.515 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:45:05.581 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:45:05.734 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:45:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:45:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:45:06.494 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:45:06.494 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:06.494 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:45:06.494 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:45:06.494 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:06.494 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:45:06.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:09 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:45:07.530 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:45:07.531 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:45:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-17T15:45:07.531 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:45:07.531 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:45:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.2 M 2024-09-17T15:45:07.531 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:07.531 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.2 M 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 14 M 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:45:08.485 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-19.3.0-4975.g1889a165.el9.x86_64.rpm 3.3 MB/s | 3.2 MB 00:00 2024-09-17T15:45:08.486 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:45:08.486 INFO:teuthology.orchestra.run.smithi064.stdout:Total 3.3 MB/s | 3.2 MB 00:00 2024-09-17T15:45:08.486 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:45:08.497 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:45:08.498 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:45:08.586 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:45:08.586 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:45:09.048 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:45:09.127 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:45:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:45:09.154 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:45:09.154 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T15:45:09.154 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:45:09.154 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:45:09.154 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:10.286 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:45:10.287 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:10.287 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:45:10.287 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:45:10.287 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:10.287 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:45:10.571 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd 2024-09-17T15:45:11.183 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:13 ago on Tue 17 Sep 2024 03:42:58 PM UTC. 2024-09-17T15:45:11.324 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-17T15:45:11.324 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:45:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-09-17T15:45:11.325 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:45:11.325 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-17T15:45:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 170 k 2024-09-17T15:45:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:11.325 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-17T15:45:11.325 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-17T15:45:11.325 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-17T15:45:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:11.325 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 170 k 2024-09-17T15:45:11.325 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 490 k 2024-09-17T15:45:11.325 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-17T15:45:11.522 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-19.3.0-4975.g1889a165.el9.x86_64.rpm 867 kB/s | 170 kB 00:00 2024-09-17T15:45:11.522 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:45:11.523 INFO:teuthology.orchestra.run.smithi064.stdout:Total 858 kB/s | 170 kB 00:00 2024-09-17T15:45:11.523 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-17T15:45:11.531 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-17T15:45:11.532 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-17T15:45:11.600 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-17T15:45:11.600 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-17T15:45:11.761 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-17T15:45:11.865 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:45:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:45:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:45:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:12.559 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-17T15:45:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:45:12.559 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:12.559 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-17T15:45:12.836 DEBUG:teuthology.parallel:result is None 2024-09-17T15:45:12.837 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:45:12.991 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T15:45:13.013 INFO:teuthology.orchestra.run.smithi064.stdout:19.3.0-4975.g1889a165.el9 2024-09-17T15:45:13.014 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975.g1889a165.el9 2024-09-17T15:45:13.014 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975.g1889a165 is installed. 2024-09-17T15:45:13.016 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:45:13.163 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T15:45:13.188 INFO:teuthology.orchestra.run.smithi100.stdout:19.3.0-4975.g1889a165.el9 2024-09-17T15:45:13.188 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975.g1889a165.el9 2024-09-17T15:45:13.188 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975.g1889a165 is installed. 2024-09-17T15:45:13.190 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T15:45:13.190 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:13.191 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T15:45:13.218 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:13.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T15:45:13.257 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T15:45:13.257 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:13.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T15:45:13.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T15:45:13.350 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:13.350 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T15:45:13.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T15:45:13.445 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T15:45:13.446 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:13.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T15:45:13.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T15:45:13.558 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:13.558 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T15:45:13.588 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T15:45:13.654 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T15:45:13.654 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:13.654 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T15:45:13.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T15:45:13.745 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:13.745 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T15:45:13.770 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T15:45:13.834 INFO:teuthology.run_tasks:Running task ceph... 2024-09-17T15:45:13.923 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-17T15:45:13.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2024-09-17T15:45:13.926 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 777 /var/log/ceph 2024-09-17T15:45:13.951 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-17T15:45:13.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T15:45:13.993 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T15:45:14.018 INFO:tasks.ceph:Creating extra log directories... 2024-09-17T15:45:14.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T15:45:14.059 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T15:45:14.092 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-17T15:45:14.093 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-17T15:45:14.093 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989', 'branch': 'wip-bharath10-testing-2024-09-17-1348', 'description': 'rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7908989', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath10-testing-2024-09-17-1348'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'mon_bind_msgr2': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath10-testing-2024-09-17-1348', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'], ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 3421, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath10-testing-2024-09-17-1348', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'targets': {'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKd61u5bTt7mzfpvYgDLEtHvBDLQTVqgMl5Ru2s78Rs5f7z0cs201ugVHdSox0HRtwVjXPRVNYFp5ebaLhS9HFw=', 'smithi100.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnbv4lSTnyEFOtpCak4U6GM24J+nK0hAZY+rZviapQ3cNKYK4+RP0/lSf43uC1XqkqjjPfjcKzTL08d+IxINn0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1, 'thrash_many': True}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_15:01:33', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-17T15:45:14.093 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T15:45:14.125 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T15:45:14.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T15:45:14.181 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T15:45:14.205 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:14.206 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T15:45:14.261 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T15:45:14.261 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-09-17T15:45:14.316 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T15:45:14.317 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:45:14.317 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-17T15:45:14.317 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:45:14.317 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:45:14.317 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-17 15:45:09.422360001 +0000 2024-09-17T15:45:14.317 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-17 15:42:17.428691431 +0000 2024-09-17T15:45:14.317 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-17 15:42:17.428691431 +0000 2024-09-17T15:45:14.317 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-17 15:42:17.428691431 +0000 2024-09-17T15:45:14.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T15:45:14.381 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-17T15:45:14.381 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-17T15:45:14.381 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000254384 s, 2.0 MB/s 2024-09-17T15:45:14.382 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T15:45:14.439 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-09-17T15:45:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T15:45:14.495 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:45:14.495 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-17T15:45:14.495 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:45:14.495 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:45:14.495 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-17 15:45:09.423359981 +0000 2024-09-17T15:45:14.495 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-17 15:42:17.768684544 +0000 2024-09-17T15:45:14.496 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-17 15:42:17.768684544 +0000 2024-09-17T15:45:14.496 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-17 15:42:17.768684544 +0000 2024-09-17T15:45:14.496 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T15:45:14.559 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-17T15:45:14.559 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-17T15:45:14.559 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000207913 s, 2.5 MB/s 2024-09-17T15:45:14.561 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T15:45:14.617 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-09-17T15:45:14.673 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T15:45:14.673 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:45:14.673 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-17T15:45:14.673 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:45:14.673 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:45:14.673 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-17 15:45:09.423359981 +0000 2024-09-17T15:45:14.673 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-17 15:42:18.224675308 +0000 2024-09-17T15:45:14.674 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-17 15:42:18.224675308 +0000 2024-09-17T15:45:14.674 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-17 15:42:18.224675308 +0000 2024-09-17T15:45:14.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T15:45:14.737 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-17T15:45:14.737 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-17T15:45:14.737 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000239706 s, 2.1 MB/s 2024-09-17T15:45:14.739 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T15:45:14.794 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-09-17T15:45:14.850 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T15:45:14.850 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:45:14.850 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-17T15:45:14.850 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:45:14.851 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:45:14.851 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-17 15:45:09.424359961 +0000 2024-09-17T15:45:14.851 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-17 15:42:18.569668320 +0000 2024-09-17T15:45:14.851 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-17 15:42:18.569668320 +0000 2024-09-17T15:45:14.851 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-17 15:42:18.569668320 +0000 2024-09-17T15:45:14.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T15:45:14.915 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-17T15:45:14.915 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-17T15:45:14.915 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000184286 s, 2.8 MB/s 2024-09-17T15:45:14.916 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T15:45:14.973 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T15:45:14.973 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:14.973 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T15:45:14.991 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T15:45:14.991 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_1 2024-09-17T15:45:15.046 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T15:45:15.046 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:45:15.046 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-17T15:45:15.047 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:45:15.047 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:45:15.047 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-17 15:44:40.687518137 +0000 2024-09-17T15:45:15.047 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-17 15:42:14.838973205 +0000 2024-09-17T15:45:15.047 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-17 15:42:14.838973205 +0000 2024-09-17T15:45:15.047 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-17 15:42:14.838973205 +0000 2024-09-17T15:45:15.047 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T15:45:15.110 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-17T15:45:15.110 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-17T15:45:15.111 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000253974 s, 2.0 MB/s 2024-09-17T15:45:15.111 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T15:45:15.168 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_2 2024-09-17T15:45:15.223 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T15:45:15.223 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:45:15.223 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-17T15:45:15.223 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:45:15.224 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:45:15.224 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-17 15:44:40.688518112 +0000 2024-09-17T15:45:15.224 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-17 15:42:15.173964661 +0000 2024-09-17T15:45:15.224 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-17 15:42:15.173964661 +0000 2024-09-17T15:45:15.224 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-17 15:42:15.173964661 +0000 2024-09-17T15:45:15.224 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T15:45:15.286 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-17T15:45:15.287 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-17T15:45:15.287 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000201837 s, 2.5 MB/s 2024-09-17T15:45:15.288 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T15:45:15.344 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_3 2024-09-17T15:45:15.399 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T15:45:15.399 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:45:15.399 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-17T15:45:15.400 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:45:15.400 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:45:15.400 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-17 15:44:40.688518112 +0000 2024-09-17T15:45:15.400 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-17 15:42:15.478956882 +0000 2024-09-17T15:45:15.400 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-17 15:42:15.478956882 +0000 2024-09-17T15:45:15.400 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-17 15:42:15.478956882 +0000 2024-09-17T15:45:15.400 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T15:45:15.463 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-17T15:45:15.463 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-17T15:45:15.463 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000266302 s, 1.9 MB/s 2024-09-17T15:45:15.464 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T15:45:15.521 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_4 2024-09-17T15:45:15.576 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T15:45:15.576 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:45:15.576 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-17T15:45:15.576 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:45:15.576 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:45:15.576 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-17 15:44:40.688518112 +0000 2024-09-17T15:45:15.576 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-17 15:42:15.798948721 +0000 2024-09-17T15:45:15.576 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-17 15:42:15.798948721 +0000 2024-09-17T15:45:15.576 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-17 15:42:15.798948721 +0000 2024-09-17T15:45:15.577 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T15:45:15.639 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-17T15:45:15.639 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-17T15:45:15.639 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000196479 s, 2.6 MB/s 2024-09-17T15:45:15.640 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T15:45:15.696 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T15:45:15.697 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-17T15:45:15.697 INFO:tasks.ceph:Generating config... 2024-09-17T15:45:15.698 INFO:tasks.ceph:[client] debug monc = 20 2024-09-17T15:45:15.698 INFO:tasks.ceph:[client] debug ms = 1 2024-09-17T15:45:15.698 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-17T15:45:15.698 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-17T15:45:15.698 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-17T15:45:15.698 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-17T15:45:15.699 INFO:tasks.ceph:[global] ms type = async 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mon] paxos min = 10 2024-09-17T15:45:15.699 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-17T15:45:15.700 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] bluestore write v2 = False 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-17T15:45:15.700 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-17T15:45:15.701 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-17T15:45:15.701 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-17T15:45:15.701 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-17T15:45:15.701 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-17T15:45:15.701 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-17T15:45:15.701 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-17T15:45:15.701 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-17T15:45:15.701 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-17T15:45:15.701 INFO:tasks.ceph:Setting up mon.a... 2024-09-17T15:45:15.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-17T15:45:15.756 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.keyring 2024-09-17T15:45:15.759 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-17T15:45:15.805 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T15:45:15.832 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.64:6789]'), ('mon.b', '[v1:172.21.15.64:6790]'), ('mon.c', '[v1:172.21.15.64:6791]'), ('mon.d', '[v1:172.21.15.64:6792]'), ('mon.e', '[v1:172.21.15.64:6793]'), ('mon.f', '[v1:172.21.15.100:6789]'), ('mon.g', '[v1:172.21.15.100:6790]'), ('mon.h', '[v1:172.21.15.100:6791]'), ('mon.i', '[v1:172.21.15.100:6792]')] 2024-09-17T15:45:15.832 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.64:6789],[v1:172.21.15.64:6790],[v1:172.21.15.64:6791],[v1:172.21.15.64:6792],[v1:172.21.15.64:6793],[v1:172.21.15.100:6789],[v1:172.21.15.100:6790],[v1:172.21.15.100:6791],[v1:172.21.15.100:6792]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mon.d': {}, 'mon.e': {}, 'mon.f': {}, 'mon.g': {}, 'mon.h': {}, 'mon.i': {}} 2024-09-17T15:45:15.833 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:15.833 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T15:45:15.888 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.64:6789]' --addv b '[v1:172.21.15.64:6790]' --addv c '[v1:172.21.15.64:6791]' --addv d '[v1:172.21.15.64:6792]' --addv e '[v1:172.21.15.64:6793]' --addv f '[v1:172.21.15.100:6789]' --addv g '[v1:172.21.15.100:6790]' --addv h '[v1:172.21.15.100:6791]' --addv i '[v1:172.21.15.100:6792]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:45:15.995 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:45:15.995 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: generated fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:15.995 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-09-17T15:45:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 0 2024-09-17T15:45:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:min_mon_release 17 (quincy) 2024-09-17T15:45:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:election_strategy: 3 2024-09-17T15:45:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:1: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:2: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:3: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:4: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:5: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:6: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:15.997 INFO:teuthology.orchestra.run.smithi064.stdout:7: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:15.997 INFO:teuthology.orchestra.run.smithi064.stdout:8: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:15.997 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (9 monitors) 2024-09-17T15:45:15.998 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T15:45:16.014 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3de99658-3189-432a-9efe-6c9d4c0a2934... 2024-09-17T15:45:16.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T15:45:16.057 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T15:45:16.099 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-09-17T15:45:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2024-09-17T15:45:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T15:45:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2024-09-17T15:45:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2024-09-17T15:45:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2024-09-17T15:45:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2024-09-17T15:45:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2024-09-17T15:45:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2024-09-17T15:45:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2024-09-17T15:45:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T15:45:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2024-09-17T15:45:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T15:45:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T15:45:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2024-09-17T15:45:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2024-09-17T15:45:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = "[v1:172.21.15.64:6789],[v1:172.21.15.64:6790],[v1:172.21.15.64:6791],[v1:172.21.15.64:6792],[v1:172.21.15.64:6793],[v1:172.21.15.100:6789],[v1:172.21.15.100:6790],[v1:172.21.15.100:6791],[v1:172.21.15.100:6792]" 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: mon client directed command retry = 5 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: mon election default strategy = 3 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: ms bind msgr2 = False 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject socket failures = 5000 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: ms type = async 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2024-09-17T15:45:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 600 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug trim objects = true 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2024-09-17T15:45:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2024-09-17T15:45:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2024-09-17T15:45:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2024-09-17T15:45:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2024-09-17T15:45:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2024-09-17T15:45:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2024-09-17T15:45:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2024-09-17T15:45:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2024-09-17T15:45:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression algorithm = zlib 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression mode = aggressive 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore write v2 = False 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore zero block detection = True 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 20 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 20 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 10 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2024-09-17T15:45:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify cached snaps = True 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify missing on start = True 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock override recovery settings = True 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue = debug_random 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue cut off = debug_random 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:45:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T15:45:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: mon min osdmap epochs = 25 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune interval = 2 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune min = 15 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune txsize = 2 2024-09-17T15:45:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: mon scrub inject crc mismatch = 0.01 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: mon scrub inject missing keys = 0.05 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: mon scrub interval = 300 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: paxos min = 10 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: paxos service trim min = 5 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: paxos trim min = 10 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2024-09-17T15:45:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-09-17T15:45:16.110 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2024-09-17T15:45:16.110 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.b] 2024-09-17T15:45:16.110 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.c] 2024-09-17T15:45:16.110 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.d] 2024-09-17T15:45:16.110 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.e] 2024-09-17T15:45:16.110 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.f] 2024-09-17T15:45:16.110 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.g] 2024-09-17T15:45:16.110 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.h] 2024-09-17T15:45:16.110 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.i] 2024-09-17T15:45:16.111 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-09-17T15:45:16.111 INFO:teuthology.orchestra.run.smithi100.stdout: chdir = "" 2024-09-17T15:45:16.111 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T15:45:16.111 INFO:teuthology.orchestra.run.smithi100.stdout: auth supported = cephx 2024-09-17T15:45:16.111 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.111 INFO:teuthology.orchestra.run.smithi100.stdout: filestore xattr use omap = true 2024-09-17T15:45:16.111 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.111 INFO:teuthology.orchestra.run.smithi100.stdout: mon clock drift allowed = 1.000 2024-09-17T15:45:16.111 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.111 INFO:teuthology.orchestra.run.smithi100.stdout: osd crush chooseleaf type = 0 2024-09-17T15:45:16.111 INFO:teuthology.orchestra.run.smithi100.stdout: auth debug = true 2024-09-17T15:45:16.111 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.111 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on old message = true 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on bug = true 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: mon pg warn max object skew = 0 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default size = 2 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow primary affinity = true 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow pg remap = true 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on legacy crush tunables = false 2024-09-17T15:45:16.112 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on crush straw calc version zero = false 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on no sortbitwise = false 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on osd down out interval zero = false 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on too few osds = false 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: mon_allow_pool_size_one = true 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: osd default data pool replay window = 5 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: mon allow pool delete = true 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: mon cluster log file level = debug 2024-09-17T15:45:16.113 INFO:teuthology.orchestra.run.smithi100.stdout: debug asserts on shutdown = true 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: mon health detail to clog = false 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: mon host = "[v1:172.21.15.64:6789],[v1:172.21.15.64:6790],[v1:172.21.15.64:6791],[v1:172.21.15.64:6792],[v1:172.21.15.64:6793],[v1:172.21.15.100:6789],[v1:172.21.15.100:6790],[v1:172.21.15.100:6791],[v1:172.21.15.100:6792]" 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: mon client directed command retry = 5 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: mon election default strategy = 3 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: ms bind msgr2 = False 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: ms inject socket failures = 5000 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: ms type = async 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout:[osd] 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: osd journal size = 100 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub load threshold = 5.0 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub max interval = 600 2024-09-17T15:45:16.114 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock profile = high_recovery_ops 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: osd recover clone overlap = true 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: osd recovery max chunk = 1048576 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug shutdown = true 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug op order = true 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify stray on activate = true 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug trim objects = true 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: osd open classes on start = true 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug pg log writeout = true 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: osd deep scrub update digest min age = 30 2024-09-17T15:45:16.115 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: osd map max advance = 10 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: journal zero on create = true 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: filestore ondisk finisher threads = 3 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: filestore apply finisher threads = 3 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: bdev debug aio = true 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug misdirected ops = true 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore block size = 96636764160 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore compression algorithm = zlib 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore compression mode = aggressive 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore fsck on mount = True 2024-09-17T15:45:16.116 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore write v2 = False 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore zero block detection = True 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluefs = 20 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluestore = 20 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: debug osd = 20 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: debug rocksdb = 10 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd full ratio = 0.9 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd nearfull ratio = 0.8 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify cached snaps = True 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify missing on start = True 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: osd failsafe full ratio = 0.95 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock override recovery settings = True 2024-09-17T15:45:16.117 INFO:teuthology.orchestra.run.smithi100.stdout: osd objectstore = bluestore 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout: osd op queue = debug_random 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout: osd op queue cut off = debug_random 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr] 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout: debug mgr = 20 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout: mgr/telemetry/nag = false 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout:[mon] 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-17T15:45:16.118 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-09-17T15:45:16.119 INFO:teuthology.orchestra.run.smithi100.stdout: debug paxos = 20 2024-09-17T15:45:16.119 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-09-17T15:45:16.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon data avail warn = 5 2024-09-17T15:45:16.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon mgr mkfs grace = 240 2024-09-17T15:45:16.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:45:16.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd reporter subtree level = osd 2024-09-17T15:45:16.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd prime pg temp = true 2024-09-17T15:45:16.119 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:45:16.119 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.119 INFO:teuthology.orchestra.run.smithi100.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T15:45:16.119 INFO:teuthology.orchestra.run.smithi100.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: # 1m isn't quite enough 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: mon_down_mkfs_grace = 2m 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_filestore_osds = false 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: mon min osdmap epochs = 25 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: mon osdmap full prune interval = 2 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: mon osdmap full prune min = 15 2024-09-17T15:45:16.120 INFO:teuthology.orchestra.run.smithi100.stdout: mon osdmap full prune txsize = 2 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: mon scrub inject crc mismatch = 0.01 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: mon scrub inject missing keys = 0.05 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: mon scrub interval = 300 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: paxos min = 10 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: paxos service trim min = 5 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: paxos trim min = 10 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout:[client] 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: rgw cache enabled = true 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable ops log = true 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable usage log = true 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T15:45:16.121 INFO:teuthology.orchestra.run.smithi100.stdout: debug monc = 20 2024-09-17T15:45:16.122 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-17T15:45:16.122 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.a] 2024-09-17T15:45:16.122 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.b] 2024-09-17T15:45:16.122 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.c] 2024-09-17T15:45:16.122 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.d] 2024-09-17T15:45:16.122 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.e] 2024-09-17T15:45:16.122 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.f] 2024-09-17T15:45:16.122 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.g] 2024-09-17T15:45:16.122 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.h] 2024-09-17T15:45:16.122 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.i] 2024-09-17T15:45:16.123 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-17T15:45:16.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-17T15:45:16.220 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-17T15:45:16.220 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:16.220 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-17T15:45:16.236 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:16.236 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-17T15:45:16.291 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2024-09-17T15:45:16.291 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:16.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T15:45:16.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T15:45:16.367 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:16.367 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:45:16.422 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T15:45:16.422 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:16.422 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T15:45:16.422 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T15:45:16.458 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:16.458 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:45:16.514 INFO:tasks.ceph:Setting up mon nodes... 2024-09-17T15:45:16.514 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-17T15:45:16.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T15:45:16.634 INFO:teuthology.orchestra.run.smithi100.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T15:45:16.637 INFO:tasks.ceph:Setting up mds nodes... 2024-09-17T15:45:16.637 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-17T15:45:16.637 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T15:45:16.678 INFO:teuthology.orchestra.run.smithi100.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-17T15:45:16.690 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-17T15:45:16.690 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-17T15:45:16.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-17T15:45:16.717 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T15:45:16.717 INFO:tasks.ceph:role: osd.0 2024-09-17T15:45:16.717 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-17T15:45:16.717 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T15:45:16.789 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:45:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:45:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:45:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T15:45:16.789 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:45:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:45:16.790 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:45:16.790 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T15:45:16.790 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:45:16.790 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:45:17.277 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-17T15:45:17.279 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-17T15:45:17.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-17T15:45:17.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-17T15:45:17.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-17T15:45:17.512 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T15:45:17.512 INFO:tasks.ceph:role: osd.1 2024-09-17T15:45:17.512 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-17T15:45:17.512 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T15:45:17.587 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:45:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:45:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:45:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T15:45:17.587 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:45:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:45:17.587 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:45:17.587 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T15:45:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:45:17.587 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:45:18.051 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-17T15:45:18.053 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-17T15:45:18.053 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-17T15:45:18.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-17T15:45:18.181 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-17T15:45:18.248 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T15:45:18.248 INFO:tasks.ceph:role: osd.2 2024-09-17T15:45:18.248 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-17T15:45:18.248 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T15:45:18.322 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:45:18.322 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:45:18.322 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:45:18.322 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T15:45:18.323 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:45:18.323 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:45:18.323 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:45:18.323 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T15:45:18.323 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:45:18.323 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:45:18.806 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-17T15:45:18.808 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-17T15:45:18.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-17T15:45:18.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-17T15:45:18.973 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:45:19.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:19.068+0000 7fe3f580d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-17T15:45:19.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:19.070+0000 7fe3f580d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-17T15:45:19.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:19.070+0000 7fe3f580d640 -1 bdev(0x55727db51800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-17T15:45:19.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:19.070+0000 7fe3f580d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-17T15:45:21.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T15:45:22.010 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:45:22.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:22.102+0000 7f7c4ddfd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-17T15:45:22.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:22.103+0000 7f7c4ddfd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-17T15:45:22.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:22.103+0000 7f7c4ddfd640 -1 bdev(0x55a86640b800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-17T15:45:22.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:22.103+0000 7f7c4ddfd640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-17T15:45:25.527 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T15:45:25.558 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:45:25.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:25.650+0000 7f78d982c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-17T15:45:25.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:25.650+0000 7f78d982c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-17T15:45:25.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:25.650+0000 7f78d982c640 -1 bdev(0x55dd0abe7800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-17T15:45:25.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:25.650+0000 7f78d982c640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-17T15:45:28.815 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T15:45:28.846 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-17T15:45:28.876 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T15:45:28.877 INFO:tasks.ceph:role: osd.3 2024-09-17T15:45:28.877 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T15:45:28.877 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T15:45:28.955 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:45:28.955 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:45:28.955 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:45:28.955 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T15:45:28.955 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:45:28.955 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:45:28.955 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:45:28.955 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T15:45:28.955 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:45:28.955 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:45:29.457 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-17T15:45:29.459 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-17T15:45:29.459 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-17T15:45:29.591 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-17T15:45:29.626 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-17T15:45:29.693 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T15:45:29.693 INFO:tasks.ceph:role: osd.4 2024-09-17T15:45:29.693 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T15:45:29.693 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T15:45:29.769 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:45:29.769 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:45:29.769 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:45:29.769 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T15:45:29.769 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:45:29.769 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:45:29.770 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:45:29.770 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T15:45:29.770 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:45:29.770 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:45:30.229 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-17T15:45:30.231 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-17T15:45:30.231 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-17T15:45:30.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-17T15:45:30.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-17T15:45:30.422 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T15:45:30.422 INFO:tasks.ceph:role: osd.5 2024-09-17T15:45:30.422 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T15:45:30.422 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T15:45:30.496 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:45:30.496 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:45:30.496 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:45:30.496 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T15:45:30.496 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:45:30.496 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:45:30.496 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:45:30.496 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T15:45:30.496 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:45:30.496 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:45:30.987 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-17T15:45:30.990 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-17T15:45:30.990 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-17T15:45:31.125 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-17T15:45:31.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:45:31.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:31.269+0000 7f6e38c0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-17T15:45:31.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:31.270+0000 7f6e38c0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-17T15:45:31.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:31.270+0000 7f6e38c0d640 -1 bdev(0x55b938d7f800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-17T15:45:31.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:31.270+0000 7f6e38c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-17T15:45:34.427 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T15:45:34.457 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:45:34.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:34.546+0000 7f5b8181e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-17T15:45:34.548 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:34.547+0000 7f5b8181e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-17T15:45:34.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:34.547+0000 7f5b8181e640 -1 bdev(0x563d6f2fb800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-17T15:45:34.549 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:34.547+0000 7f5b8181e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-17T15:45:37.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T15:45:37.497 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:45:37.588 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:37.586+0000 7fc4cdeda640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-17T15:45:37.589 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:37.587+0000 7fc4cdeda640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-17T15:45:37.589 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:37.587+0000 7fc4cdeda640 -1 bdev(0x55c248759800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-17T15:45:37.589 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:37.587+0000 7fc4cdeda640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-17T15:45:40.511 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T15:45:40.540 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-17T15:45:40.541 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:40.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-17T15:45:40.571 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:40.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-17T15:45:40.642 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:40.642 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-17T15:45:40.712 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:40.712 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-17T15:45:40.741 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:40.741 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-17T15:45:40.808 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:40.809 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-17T15:45:40.876 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:40.876 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-17T15:45:40.942 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:40.942 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-17T15:45:40.998 INFO:tasks.ceph:Adding keys to all mons... 2024-09-17T15:45:40.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T15:45:41.001 DEBUG:teuthology.orchestra.run.smithi100:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T15:45:41.041 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2024-09-17T15:45:41.041 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAPpOlmOTMpBBAAPa/xm2gjY9qCbXVKV2e5Bg== 2024-09-17T15:45:41.041 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2024-09-17T15:45:41.041 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQASpOlmpz4tBhAAXtvlvRBPdASkLK8+xiz+iQ== 2024-09-17T15:45:41.041 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2024-09-17T15:45:41.041 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAVpOlmLtDRJhAAbSWzpE9A+xTryfXDU+N+4w== 2024-09-17T15:45:41.042 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2024-09-17T15:45:41.042 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAMpOlm8cnJJRAArJmWByIY+uzs69Sb/Uyyvw== 2024-09-17T15:45:41.042 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2024-09-17T15:45:41.042 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAbpOlmBWkaEBAA2VrIhuY7Nt14kPQZyKqFsg== 2024-09-17T15:45:41.042 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.4] 2024-09-17T15:45:41.042 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAepOlmLzGeIBAAoqQIKOZlW+LvRZbogF3l3g== 2024-09-17T15:45:41.042 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.5] 2024-09-17T15:45:41.042 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAhpOlm38MAIxAAHI5W+6y/5ExrjlWAyY7pIA== 2024-09-17T15:45:41.042 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-09-17T15:45:41.042 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAMpOlmMVtkKBAAOniSzE5ksb/v4qkl6ON+SQ== 2024-09-17T15:45:41.062 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.0] 2024-09-17T15:45:41.062 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQAPpOlmOTMpBBAAPa/xm2gjY9qCbXVKV2e5Bg== 2024-09-17T15:45:41.062 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.1] 2024-09-17T15:45:41.062 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQASpOlmpz4tBhAAXtvlvRBPdASkLK8+xiz+iQ== 2024-09-17T15:45:41.062 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.2] 2024-09-17T15:45:41.062 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQAVpOlmLtDRJhAAbSWzpE9A+xTryfXDU+N+4w== 2024-09-17T15:45:41.063 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr.x] 2024-09-17T15:45:41.063 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQAMpOlm8cnJJRAArJmWByIY+uzs69Sb/Uyyvw== 2024-09-17T15:45:41.063 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.3] 2024-09-17T15:45:41.063 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQAbpOlmBWkaEBAA2VrIhuY7Nt14kPQZyKqFsg== 2024-09-17T15:45:41.063 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.4] 2024-09-17T15:45:41.063 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQAepOlmLzGeIBAAoqQIKOZlW+LvRZbogF3l3g== 2024-09-17T15:45:41.063 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.5] 2024-09-17T15:45:41.063 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQAhpOlm38MAIxAAHI5W+6y/5ExrjlWAyY7pIA== 2024-09-17T15:45:41.063 INFO:teuthology.orchestra.run.smithi100.stdout:[client.0] 2024-09-17T15:45:41.063 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQAMpOlmMVtkKBAAOniSzE5ksb/v4qkl6ON+SQ== 2024-09-17T15:45:41.064 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:45:41.088 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:45:41.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:45:41.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:45:41.214 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:45:41.216 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:45:41.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T15:45:41.270 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T15:45:41.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:45:41.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:45:41.374 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:45:41.376 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:45:41.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:45:41.430 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:45:41.481 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T15:45:41.484 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T15:45:41.538 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-17T15:45:41.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-17T15:45:41.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T15:45:41.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-17T15:45:41.687 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-17T15:45:41.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T15:45:41.853 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-17T15:45:41.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-17T15:45:41.946 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T15:45:42.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-17T15:45:42.070 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-d 2024-09-17T15:45:42.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i d --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T15:45:42.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-d 2024-09-17T15:45:42.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-e 2024-09-17T15:45:42.325 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i e --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T15:45:42.422 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-e 2024-09-17T15:45:42.449 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-f 2024-09-17T15:45:42.478 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i f --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T15:45:42.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-f 2024-09-17T15:45:42.607 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-g 2024-09-17T15:45:42.672 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i g --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T15:45:42.768 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-g 2024-09-17T15:45:42.793 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-h 2024-09-17T15:45:42.858 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i h --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T15:45:42.953 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-h 2024-09-17T15:45:42.977 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-i 2024-09-17T15:45:43.042 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i i --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T15:45:43.140 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-i 2024-09-17T15:45:43.164 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:45:43.168 DEBUG:teuthology.orchestra.run.smithi100:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:45:43.220 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-17T15:45:43.220 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T15:45:43.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T15:45:43.226 INFO:tasks.ceph.mon.a:Started 2024-09-17T15:45:43.226 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:45:43.226 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:45:43.229 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:45:43.229 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T15:45:43.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T15:45:43.231 INFO:tasks.ceph.mon.c:Started 2024-09-17T15:45:43.231 INFO:tasks.ceph.mon.d:Restarting daemon 2024-09-17T15:45:43.231 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-09-17T15:45:43.234 INFO:tasks.ceph.mon.d:Started 2024-09-17T15:45:43.234 INFO:tasks.ceph.mon.e:Restarting daemon 2024-09-17T15:45:43.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-09-17T15:45:43.237 INFO:tasks.ceph.mon.e:Started 2024-09-17T15:45:43.237 INFO:tasks.ceph.mon.f:Restarting daemon 2024-09-17T15:45:43.237 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-09-17T15:45:43.262 INFO:tasks.ceph.mon.f:Started 2024-09-17T15:45:43.262 INFO:tasks.ceph.mon.g:Restarting daemon 2024-09-17T15:45:43.262 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-09-17T15:45:43.264 INFO:tasks.ceph.mon.g:Started 2024-09-17T15:45:43.264 INFO:tasks.ceph.mon.h:Restarting daemon 2024-09-17T15:45:43.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-09-17T15:45:43.266 INFO:tasks.ceph.mon.h:Started 2024-09-17T15:45:43.267 INFO:tasks.ceph.mon.i:Restarting daemon 2024-09-17T15:45:43.267 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-09-17T15:45:43.269 INFO:tasks.ceph.mon.i:Started 2024-09-17T15:45:43.269 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-17T15:45:43.269 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-17T15:45:43.269 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-17T15:45:43.271 INFO:tasks.ceph.mgr.x:Started 2024-09-17T15:45:43.271 DEBUG:tasks.ceph:set 0 configs 2024-09-17T15:45:43.272 INFO:tasks.ceph:Setting crush tunables to default 2024-09-17T15:45:43.272 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd crush tunables default 2024-09-17T15:45:43.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.397+0000 7f3286189640 10 monclient: get_monmap_and_config 2024-09-17T15:45:43.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.397+0000 7f3286189640 10 monclient: build_initial_monmap 2024-09-17T15:45:43.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.397+0000 7f3286189640 10 monclient: monmap: 2024-09-17T15:45:43.399 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:45:43.399 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:43.398572+0000 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:43.398572+0000 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:43.400 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:43.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.398+0000 7f3286189640 1 Processor -- start 2024-09-17T15:45:43.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.398+0000 7f3286189640 1 -- start start 2024-09-17T15:45:43.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.398+0000 7f3286189640 10 monclient: init 2024-09-17T15:45:43.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.398+0000 7f3286189640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:43.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.398+0000 7f3286189640 10 monclient: _add_conns ranks=[4,3,1,0,2,8,7,6,5] 2024-09-17T15:45:43.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.399+0000 7f3286189640 10 monclient(hunting): picked mon.noname-c con 0x7f32800a5600 addr v1:172.21.15.64:6791/0 2024-09-17T15:45:43.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.399+0000 7f3286189640 10 monclient(hunting): picked mon.noname-g con 0x7f328009ccc0 addr v1:172.21.15.100:6790/0 2024-09-17T15:45:43.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.399+0000 7f3286189640 10 monclient(hunting): picked mon.noname-f con 0x7f328013f720 addr v1:172.21.15.100:6789/0 2024-09-17T15:45:43.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.399+0000 7f3286189640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3280139140 con 0x7f328013f720 2024-09-17T15:45:43.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.399+0000 7f3286189640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f328009d7f0 con 0x7f328009ccc0 2024-09-17T15:45:43.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.399+0000 7f3286189640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f328009dad0 con 0x7f32800a5600 2024-09-17T15:45:43.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.399+0000 7f3286189640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:43.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.399+0000 7f3286189640 10 monclient(hunting): authenticate will time out at 741.074086s 2024-09-17T15:45:43.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.399+0000 7f327f7fe640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f32800a5600 0x7f32800a59e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:58898/0 (socket says 172.21.15.64:58898) 2024-09-17T15:45:43.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.399+0000 7f327f7fe640 1 -- 172.21.15.64:0/3066867352 learned_addr learned my addr 172.21.15.64:0/3066867352 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:45:43.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 1 -- 172.21.15.64:0/3066867352 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 778167429 0 0) 0x7f3280139140 con 0x7f328013f720 2024-09-17T15:45:43.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:43.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:43.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:43.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient(hunting): my global_id is 4105 2024-09-17T15:45:43.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 1 -- 172.21.15.64:0/3066867352 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3268003520 con 0x7f328013f720 2024-09-17T15:45:43.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 1 -- 172.21.15.64:0/3066867352 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2365273438 0 0) 0x7f328009d7f0 con 0x7f328009ccc0 2024-09-17T15:45:43.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:43.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:43.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient(hunting): my global_id is 4107 2024-09-17T15:45:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 1 -- 172.21.15.64:0/3066867352 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3280139140 con 0x7f328009ccc0 2024-09-17T15:45:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 1 -- 172.21.15.64:0/3066867352 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2279203561 0 0) 0x7f3268003520 con 0x7f328013f720 2024-09-17T15:45:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 1 -- 172.21.15.64:0/3066867352 >> v1:172.21.15.64:6791/0 conn(0x7f32800a5600 legacy=0x7f32800a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 1 -- 172.21.15.64:0/3066867352 >> v1:172.21.15.100:6790/0 conn(0x7f328009ccc0 legacy=0x7f328013e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:45:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 1 monclient: found mon.noname-f 2024-09-17T15:45:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.100:6789/0 2024-09-17T15:45:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 1 -- 172.21.15.64:0/3066867352 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328009ddb0 con 0x7f328013f720 2024-09-17T15:45:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:45:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:45:43.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 1 -- 172.21.15.64:0/3066867352 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3278001bd0 con 0x7f328013f720 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient: dump: 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:43.623 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:43.624 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:43.624 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:43.624 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:43.624 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:43.624 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:43.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f3286189640 5 monclient: authenticate success, global_id 4105 2024-09-17T15:45:43.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f3286189640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:43.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 1 -- 172.21.15.64:0/3066867352 <== mon.1 v1:172.21.15.100:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3278001df0 con 0x7f328013f720 2024-09-17T15:45:43.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:43.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 1 -- 172.21.15.64:0/3066867352 <== mon.1 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3278003cd0 con 0x7f328013f720 2024-09-17T15:45:43.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f3286189640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:43.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f3286189640 4 set_mon_vals no callback set 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.617+0000 7f327e7fc640 10 monclient: dump: 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:43.625 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:43.626 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:43.626 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:43.626 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:43.626 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:43.626 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:43.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.618+0000 7f3286189640 10 monclient: shutdown 2024-09-17T15:45:43.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.618+0000 7f3286189640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:43.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.618+0000 7f3286189640 1 -- 172.21.15.64:0/3066867352 >> v1:172.21.15.100:6789/0 conn(0x7f328013f720 legacy=0x7f3280141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:43.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.618+0000 7f3286189640 1 -- 172.21.15.64:0/3066867352 shutdown_connections 2024-09-17T15:45:43.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.618+0000 7f3286189640 1 -- 172.21.15.64:0/3066867352 >> 172.21.15.64:0/3066867352 conn(0x7f328006db40 msgr2=0x7f32800aba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:43.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.618+0000 7f3286189640 1 -- 172.21.15.64:0/3066867352 shutdown_connections 2024-09-17T15:45:43.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.618+0000 7f3286189640 1 -- 172.21.15.64:0/3066867352 wait complete. 2024-09-17T15:45:43.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.619+0000 7f3286189640 10 monclient: build_initial_monmap 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.619+0000 7f3286189640 10 monclient: monmap: 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:43.627 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:43.628 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:43.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.619+0000 7f3286189640 1 Processor -- start 2024-09-17T15:45:43.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f3286189640 1 -- start start 2024-09-17T15:45:43.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f3286189640 10 monclient: init 2024-09-17T15:45:43.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f3286189640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:43.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f3286189640 10 monclient: _add_conns ranks=[7,1,6,8,4,0,2,5,3] 2024-09-17T15:45:43.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f3286189640 10 monclient(hunting): picked mon.noname-i con 0x7f328009ccc0 addr v1:172.21.15.100:6792/0 2024-09-17T15:45:43.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f3286189640 10 monclient(hunting): picked mon.noname-f con 0x7f32800a5600 addr v1:172.21.15.100:6789/0 2024-09-17T15:45:43.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f3286189640 10 monclient(hunting): picked mon.noname-d con 0x7f328013f720 addr v1:172.21.15.64:6792/0 2024-09-17T15:45:43.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f3286189640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f328013e970 con 0x7f32800a5600 2024-09-17T15:45:43.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f3286189640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f328013f4e0 con 0x7f328013f720 2024-09-17T15:45:43.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f3286189640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3280138b10 con 0x7f328009ccc0 2024-09-17T15:45:43.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f3286189640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:43.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f3286189640 10 monclient(hunting): authenticate will time out at 741.295847s 2024-09-17T15:45:43.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f327ffff640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f328013f720 0x7f32801d5d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:48788/0 (socket says 172.21.15.64:48788) 2024-09-17T15:45:43.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.620+0000 7f327ffff640 1 -- 172.21.15.64:0/4091480534 learned_addr learned my addr 172.21.15.64:0/4091480534 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:45:43.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4292270870 0 0) 0x7f328009ddb0 con 0x7f328013f720 2024-09-17T15:45:43.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): _negotiate 2024-09-17T15:45:43.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:43.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:43.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): my global_id is 4110 2024-09-17T15:45:43.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f326c003550 con 0x7f328013f720 2024-09-17T15:45:43.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1083092868 0 0) 0x7f328013e970 con 0x7f32800a5600 2024-09-17T15:45:43.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): _negotiate 2024-09-17T15:45:43.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): my global_id is 4123 2024-09-17T15:45:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f328009ddb0 con 0x7f32800a5600 2024-09-17T15:45:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3498572999 0 0) 0x7f328009dad0 con 0x7f328009ccc0 2024-09-17T15:45:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): _negotiate 2024-09-17T15:45:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): my global_id is 4111 2024-09-17T15:45:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f328013e970 con 0x7f328009ccc0 2024-09-17T15:45:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1556174846 0 0) 0x7f326c003550 con 0x7f328013f720 2024-09-17T15:45:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f328009dad0 con 0x7f328013f720 2024-09-17T15:45:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3278003cb0 con 0x7f328013f720 2024-09-17T15:45:43.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:45:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): dump: 2024-09-17T15:45:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:45:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:43.636 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:43.636 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:43.636 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:43.636 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:43.636 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:43.636 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:43.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:45:43.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 53652737 0 0) 0x7f328009ddb0 con 0x7f32800a5600 2024-09-17T15:45:43.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f326c003550 con 0x7f32800a5600 2024-09-17T15:45:43.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1281314777 0 0) 0x7f328013e970 con 0x7f328009ccc0 2024-09-17T15:45:43.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f328009ddb0 con 0x7f328009ccc0 2024-09-17T15:45:43.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f32740014b0 con 0x7f32800a5600 2024-09-17T15:45:43.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:43.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:45:43.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): dump: 2024-09-17T15:45:43.637 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:45:43.637 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:43.637 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:43.637 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:43.638 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:43.638 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:43.638 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:43.638 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:43.638 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:43.638 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:43.638 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:43.638 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:43.638 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:43.638 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:43.638 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:43.638 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:43.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3270001920 con 0x7f328009ccc0 2024-09-17T15:45:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:45:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient(hunting): dump: 2024-09-17T15:45:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:45:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:43.640 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2889639374 0 0) 0x7f328009dad0 con 0x7f328013f720 2024-09-17T15:45:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 >> v1:172.21.15.100:6792/0 conn(0x7f328009ccc0 legacy=0x7f328009e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 >> v1:172.21.15.100:6789/0 conn(0x7f32800a5600 legacy=0x7f328009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient: _finish_hunting 0 2024-09-17T15:45:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 monclient: found mon.d 2024-09-17T15:45:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:43.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:45:43.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32800a0890 con 0x7f328013f720 2024-09-17T15:45:43.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient: _finish_auth 0 2024-09-17T15:45:43.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient: _check_auth_tickets 2024-09-17T15:45:43.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:43.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3286189640 5 monclient: authenticate success, global_id 4110 2024-09-17T15:45:43.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3286189640 10 monclient: _renew_subs 2024-09-17T15:45:43.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3286189640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:45:43.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3286189640 1 -- 172.21.15.64:0/4091480534 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f32800a0b70 con 0x7f328013f720 2024-09-17T15:45:43.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3278003ed0 con 0x7f328013f720 2024-09-17T15:45:43.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:43.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3278004500 con 0x7f328013f720 2024-09-17T15:45:43.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:45:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 10 monclient: dump: 2024-09-17T15:45:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:45:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3286189640 10 monclient: _renew_subs 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f327dffb640 4 set_mon_vals no callback set 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3278005e90 con 0x7f328013f720 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3286189640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.621+0000 7f3286189640 1 -- 172.21.15.64:0/4091480534 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f32800a0280 con 0x7f328013f720 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.622+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3108653439 0 0) 0x7f3278004760 con 0x7f328013f720 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.623+0000 7f3286189640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.623+0000 7f3286189640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:43.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.623+0000 7f3286189640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:45:43.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.623+0000 7f3286189640 1 -- 172.21.15.64:0/4091480534 --> v1:172.21.15.64:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32800a0f10 con 0x7f328013f720 2024-09-17T15:45:43.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.626+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.6 v1:172.21.15.64:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f328013f4e0 con 0x7f328013f720 2024-09-17T15:45:43.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.626+0000 7f3267fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:43.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.626+0000 7f3267fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.702+0000 7f3286189640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-17T15:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.702+0000 7f3286189640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-17T15:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.702+0000 7f3286189640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.702+0000 7f3286189640 1 -- 172.21.15.64:0/4091480534 --> v1:172.21.15.64:6792/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f3280068be0 con 0x7f328013f720 2024-09-17T15:45:43.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.705+0000 7f3267fff640 1 -- 172.21.15.64:0/4091480534 <== mon.6 v1:172.21.15.64:6792/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (unknown 2277712955 0 0) 0x7f32780065a0 con 0x7f328013f720 2024-09-17T15:45:43.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.705+0000 7f3267fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-17T15:45:43.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.705+0000 7f3267fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-17T15:45:43.707 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-09-17T15:45:43.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.706+0000 7f3286189640 10 monclient: shutdown 2024-09-17T15:45:43.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.706+0000 7f3286189640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:43.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.706+0000 7f3286189640 1 -- 172.21.15.64:0/4091480534 >> v1:172.21.15.64:6792/0 conn(0x7f328013f720 legacy=0x7f32801d5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:43.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.706+0000 7f3286189640 1 -- 172.21.15.64:0/4091480534 shutdown_connections 2024-09-17T15:45:43.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.706+0000 7f3286189640 1 -- 172.21.15.64:0/4091480534 >> 172.21.15.64:0/4091480534 conn(0x7f328006db40 msgr2=0x7f3280137160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:43.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.707+0000 7f3286189640 1 -- 172.21.15.64:0/4091480534 shutdown_connections 2024-09-17T15:45:43.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:43.707+0000 7f3286189640 1 -- 172.21.15.64:0/4091480534 wait complete. 2024-09-17T15:45:43.718 INFO:tasks.ceph:check_enable_crimson: False 2024-09-17T15:45:43.718 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-17T15:45:43.719 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:43.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-17T15:45:43.745 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:43.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-17T15:45:43.810 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-17T15:45:43.810 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-17T15:45:43.876 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:43.876 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-17T15:45:43.901 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:43.902 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-17T15:45:43.966 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T15:45:43.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-17T15:45:44.032 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 1d9b48f9-b24c-453f-a5e9-79d74f56118c 0 2024-09-17T15:45:44.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.167+0000 7f8a0a885640 10 monclient: get_monmap_and_config 2024-09-17T15:45:44.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.167+0000 7f8a0a885640 10 monclient: build_initial_monmap 2024-09-17T15:45:44.170 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.168+0000 7f8a0a885640 10 monclient: monmap: 2024-09-17T15:45:44.170 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-17T15:45:44.170 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.171 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:44.169360+0000 2024-09-17T15:45:44.171 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:44.169360+0000 2024-09-17T15:45:44.171 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:44.171 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.171 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:44.171 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:44.171 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:44.171 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:44.172 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:44.172 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:44.172 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:44.172 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:44.172 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:44.172 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.182 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.168+0000 7f8a0a885640 1 Processor -- start 2024-09-17T15:45:44.182 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.168+0000 7f8a0a885640 1 -- start start 2024-09-17T15:45:44.182 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.168+0000 7f8a0a885640 10 monclient: init 2024-09-17T15:45:44.182 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a0a885640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:44.182 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a0a885640 10 monclient: _add_conns ranks=[2,7,1,4,8,5,6,0,3] 2024-09-17T15:45:44.183 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a0a885640 10 monclient(hunting): picked mon.noname-b con 0x7f8a040a5600 addr v1:172.21.15.64:6790/0 2024-09-17T15:45:44.183 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a0a885640 10 monclient(hunting): picked mon.noname-i con 0x7f8a041397e0 addr v1:172.21.15.100:6792/0 2024-09-17T15:45:44.183 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a0a885640 10 monclient(hunting): picked mon.noname-f con 0x7f8a0413f800 addr v1:172.21.15.100:6789/0 2024-09-17T15:45:44.183 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a0a885640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a0409dab0 con 0x7f8a0413f800 2024-09-17T15:45:44.183 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a0a885640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a0409dd90 con 0x7f8a040a5600 2024-09-17T15:45:44.183 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a0a885640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a0409e070 con 0x7f8a041397e0 2024-09-17T15:45:44.183 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a0a885640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:44.183 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a0a885640 10 monclient(hunting): authenticate will time out at 755.573468s 2024-09-17T15:45:44.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f89fbfff640 1 --1- >> v1:172.21.15.100:6792/0 conn(0x7f8a041397e0 0x7f8a0413e0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6792/0 says I am v1:172.21.15.100:45466/0 (socket says 172.21.15.100:45466) 2024-09-17T15:45:44.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f89fbfff640 1 -- 172.21.15.100:0/2333781124 learned_addr learned my addr 172.21.15.100:0/2333781124 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-17T15:45:44.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a08e22640 1 --1- 172.21.15.100:0/2333781124 >> v1:172.21.15.100:6789/0 conn(0x7f8a0413f800 0x7f8a04141bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:41172/0 (socket says 172.21.15.100:41172) 2024-09-17T15:45:44.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a037fe640 1 -- 172.21.15.100:0/2333781124 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1808670720 0 0) 0x7f89fc001490 con 0x7f8a0413f800 2024-09-17T15:45:44.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a037fe640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a037fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a037fe640 10 monclient(hunting): my global_id is 4132 2024-09-17T15:45:44.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a037fe640 1 -- 172.21.15.100:0/2333781124 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89f0003520 con 0x7f8a0413f800 2024-09-17T15:45:44.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a037fe640 1 -- 172.21.15.100:0/2333781124 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1751718690 0 0) 0x7f8a0409e070 con 0x7f8a041397e0 2024-09-17T15:45:44.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a037fe640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a037fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a037fe640 10 monclient(hunting): my global_id is 4120 2024-09-17T15:45:44.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.169+0000 7f8a037fe640 1 -- 172.21.15.100:0/2333781124 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89fc001490 con 0x7f8a041397e0 2024-09-17T15:45:44.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 1 -- 172.21.15.100:0/2333781124 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1310091330 0 0) 0x7f8a0409dd90 con 0x7f8a040a5600 2024-09-17T15:45:44.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient(hunting): my global_id is 4106 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 1 -- 172.21.15.100:0/2333781124 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a0409e070 con 0x7f8a040a5600 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 1 -- 172.21.15.100:0/2333781124 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4279853920 0 0) 0x7f89f0003520 con 0x7f8a0413f800 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 1 -- 172.21.15.100:0/2333781124 >> v1:172.21.15.100:6792/0 conn(0x7f8a041397e0 legacy=0x7f8a0413e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 1 -- 172.21.15.100:0/2333781124 >> v1:172.21.15.64:6790/0 conn(0x7f8a040a5600 legacy=0x7f8a040a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient: _finish_hunting 0 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 1 monclient: found mon.noname-f 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.100:6789/0 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 1 -- 172.21.15.100:0/2333781124 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0409ca00 con 0x7f8a0413f800 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient: _finish_auth 0 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient: _check_auth_tickets 2024-09-17T15:45:44.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 1 -- 172.21.15.100:0/2333781124 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f89fc001d80 con 0x7f8a0413f800 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient: dump: 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.188 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a0a885640 5 monclient: authenticate success, global_id 4132 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a0a885640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 1 -- 172.21.15.100:0/2333781124 <== mon.1 v1:172.21.15.100:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89fc001f20 con 0x7f8a0413f800 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 1 -- 172.21.15.100:0/2333781124 <== mon.1 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f89fc004ea0 con 0x7f8a0413f800 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a0a885640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a0a885640 4 set_mon_vals no callback set 2024-09-17T15:45:44.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a037fe640 10 monclient: dump: 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.190 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a0a885640 10 monclient: shutdown 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a0a885640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a0a885640 1 -- 172.21.15.100:0/2333781124 >> v1:172.21.15.100:6789/0 conn(0x7f8a0413f800 legacy=0x7f8a04141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a0a885640 1 -- 172.21.15.100:0/2333781124 shutdown_connections 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a0a885640 1 -- 172.21.15.100:0/2333781124 >> 172.21.15.100:0/2333781124 conn(0x7f8a0406dd40 msgr2=0x7f8a040aba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.170+0000 7f8a0a885640 1 -- 172.21.15.100:0/2333781124 shutdown_connections 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.171+0000 7f8a0a885640 1 -- 172.21.15.100:0/2333781124 wait complete. 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.171+0000 7f8a0a885640 10 monclient: build_initial_monmap 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.171+0000 7f8a0a885640 10 monclient: monmap: 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-17T15:45:44.191 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:44.192 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.171+0000 7f8a0a885640 1 Processor -- start 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.171+0000 7f8a0a885640 1 -- start start 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.171+0000 7f8a0a885640 10 monclient: init 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a0a885640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a0a885640 10 monclient: _add_conns ranks=[8,4,6,7,2,0,5,3,1] 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a0a885640 10 monclient(hunting): picked mon.noname-e con 0x7f8a040a5600 addr v1:172.21.15.64:6793/0 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a0a885640 10 monclient(hunting): picked mon.noname-c con 0x7f8a041397e0 addr v1:172.21.15.64:6791/0 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a0a885640 10 monclient(hunting): picked mon.noname-d con 0x7f8a0413f800 addr v1:172.21.15.64:6792/0 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a0a885640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a0409f1f0 con 0x7f8a041397e0 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a0a885640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a0413d820 con 0x7f8a0413f800 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a0a885640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a0413cd10 con 0x7f8a040a5600 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a0a885640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a0a885640 10 monclient(hunting): authenticate will time out at 755.576399s 2024-09-17T15:45:44.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f89fbfff640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f8a041397e0 0x7f8a0413c600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.100:45938/0 (socket says 172.21.15.100:45938) 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f89fbfff640 1 -- 172.21.15.100:0/1286503472 learned_addr learned my addr 172.21.15.100:0/1286503472 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a08e22640 1 --1- 172.21.15.100:0/1286503472 >> v1:172.21.15.64:6792/0 conn(0x7f8a0413f800 0x7f8a041d5ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.100:57240/0 (socket says 172.21.15.100:57240) 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a03fff640 1 --1- 172.21.15.100:0/1286503472 >> v1:172.21.15.64:6793/0 conn(0x7f8a040a5600 0x7f8a0413bef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.100:34498/0 (socket says 172.21.15.100:34498) 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1103003475 0 0) 0x7f8a0409f1f0 con 0x7f8a041397e0 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a017fa640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a017fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.172+0000 7f8a017fa640 10 monclient(hunting): my global_id is 4117 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89e4003520 con 0x7f8a041397e0 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 469285793 0 0) 0x7f8a0413d820 con 0x7f8a0413f800 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): my global_id is 4119 2024-09-17T15:45:44.194 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a0409f1f0 con 0x7f8a0413f800 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 564842515 0 0) 0x7f8a0413cd10 con 0x7f8a040a5600 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): my global_id is 4112 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a0413d820 con 0x7f8a040a5600 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1879506627 0 0) 0x7f89e4003520 con 0x7f8a041397e0 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8a0413cd10 con 0x7f8a041397e0 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f89ec001bb0 con 0x7f8a041397e0 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): dump: 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.195 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2431619211 0 0) 0x7f8a0413d820 con 0x7f8a040a5600 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89e4003520 con 0x7f8a040a5600 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2431891173 0 0) 0x7f8a0409f1f0 con 0x7f8a0413f800 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8a0413d820 con 0x7f8a0413f800 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f89f4001a50 con 0x7f8a040a5600 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): dump: 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.197 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f89fc001c40 con 0x7f8a0413f800 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient(hunting): dump: 2024-09-17T15:45:44.198 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.199 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3434204062 0 0) 0x7f8a0413cd10 con 0x7f8a041397e0 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 >> v1:172.21.15.64:6793/0 conn(0x7f8a040a5600 legacy=0x7f8a0413bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 >> v1:172.21.15.64:6792/0 conn(0x7f8a0413f800 legacy=0x7f8a041d5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient: _finish_hunting 0 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 monclient: found mon.c 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0409e950 con 0x7f8a041397e0 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient: _finish_auth 0 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient: _check_auth_tickets 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a0a885640 5 monclient: authenticate success, global_id 4117 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a0a885640 10 monclient: _renew_subs 2024-09-17T15:45:44.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a0a885640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a0a885640 1 -- 172.21.15.100:0/1286503472 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0409ec30 con 0x7f8a041397e0 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a0a885640 10 monclient: _renew_subs 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a0a885640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a0a885640 1 -- 172.21.15.100:0/1286503472 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f8a0413e380 con 0x7f8a041397e0 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89ec004a90 con 0x7f8a041397e0 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a02ffd640 4 set_mon_vals no callback set 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f89ec004f80 con 0x7f8a041397e0 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 10 monclient: dump: 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.201 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f89ec0054b0 con 0x7f8a041397e0 2024-09-17T15:45:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.173+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (unknown 3639480041 0 0) 0x7f89ec006a70 con 0x7f8a041397e0 2024-09-17T15:45:44.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.174+0000 7f8a0a885640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.174+0000 7f8a0a885640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.174+0000 7f8a0a885640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:44.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.174+0000 7f8a0a885640 1 -- 172.21.15.100:0/1286503472 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89e8005080 con 0x7f8a041397e0 2024-09-17T15:45:44.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.176+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f89ec005930 con 0x7f8a041397e0 2024-09-17T15:45:44.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.176+0000 7f8a017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.176+0000 7f8a017fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:44.249 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.247+0000 7f8a0a885640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1d9b48f9-b24c-453f-a5e9-79d74f56118c", "id": 0}] 2024-09-17T15:45:44.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.247+0000 7f8a0a885640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1d9b48f9-b24c-453f-a5e9-79d74f56118c", "id": 0}] 2024-09-17T15:45:44.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.247+0000 7f8a0a885640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:44.250 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.247+0000 7f8a0a885640 1 -- 172.21.15.100:0/1286503472 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "osd new", "uuid": "1d9b48f9-b24c-453f-a5e9-79d74f56118c", "id": 0} v 0) -- 0x7f89e8005370 con 0x7f8a041397e0 2024-09-17T15:45:44.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.252+0000 7f8a017fa640 1 -- 172.21.15.100:0/1286503472 <== mon.4 v1:172.21.15.64:6791/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1d9b48f9-b24c-453f-a5e9-79d74f56118c", "id": 0}]=0 v3) ==== 112+0+2 (unknown 3977349658 0 732707515) 0x7f89ec005b90 con 0x7f8a041397e0 2024-09-17T15:45:44.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.252+0000 7f8a017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1d9b48f9-b24c-453f-a5e9-79d74f56118c", "id": 0}] 2024-09-17T15:45:44.254 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.252+0000 7f8a017fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:44.255 INFO:teuthology.orchestra.run.smithi100.stdout:0 2024-09-17T15:45:44.256 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.253+0000 7f8a0a885640 10 monclient: shutdown 2024-09-17T15:45:44.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.253+0000 7f8a0a885640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:44.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.253+0000 7f8a0a885640 1 -- 172.21.15.100:0/1286503472 >> v1:172.21.15.64:6791/0 conn(0x7f8a041397e0 legacy=0x7f8a0413c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.253+0000 7f8a0a885640 1 -- 172.21.15.100:0/1286503472 shutdown_connections 2024-09-17T15:45:44.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.253+0000 7f8a0a885640 1 -- 172.21.15.100:0/1286503472 >> 172.21.15.100:0/1286503472 conn(0x7f8a0406dd40 msgr2=0x7f8a041356f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:44.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.253+0000 7f8a0a885640 1 -- 172.21.15.100:0/1286503472 shutdown_connections 2024-09-17T15:45:44.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.253+0000 7f8a0a885640 1 -- 172.21.15.100:0/1286503472 wait complete. 2024-09-17T15:45:44.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 6b8dcb8b-604c-46d3-a682-5ef3e93155d0 1 2024-09-17T15:45:44.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.361+0000 7f2d88650640 10 monclient: get_monmap_and_config 2024-09-17T15:45:44.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.361+0000 7f2d88650640 10 monclient: build_initial_monmap 2024-09-17T15:45:44.379 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.361+0000 7f2d88650640 10 monclient: monmap: 2024-09-17T15:45:44.380 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-17T15:45:44.380 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.380 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:44.362571+0000 2024-09-17T15:45:44.381 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:44.362571+0000 2024-09-17T15:45:44.381 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:44.381 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.381 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:44.381 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:44.381 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:44.382 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:44.382 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:44.382 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:44.382 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:44.382 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:44.382 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:44.383 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.361+0000 7f2d88650640 1 Processor -- start 2024-09-17T15:45:44.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.361+0000 7f2d88650640 1 -- start start 2024-09-17T15:45:44.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.361+0000 7f2d88650640 10 monclient: init 2024-09-17T15:45:44.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d88650640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:44.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d88650640 10 monclient: _add_conns ranks=[1,3,0,2,6,7,4,5,8] 2024-09-17T15:45:44.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d88650640 10 monclient(hunting): picked mon.noname-f con 0x7f2d800a5600 addr v1:172.21.15.100:6789/0 2024-09-17T15:45:44.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d88650640 10 monclient(hunting): picked mon.noname-g con 0x7f2d8009ccc0 addr v1:172.21.15.100:6790/0 2024-09-17T15:45:44.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d88650640 10 monclient(hunting): picked mon.noname-a con 0x7f2d8013f720 addr v1:172.21.15.64:6789/0 2024-09-17T15:45:44.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d88650640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d80139140 con 0x7f2d8013f720 2024-09-17T15:45:44.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d88650640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d8009d7f0 con 0x7f2d800a5600 2024-09-17T15:45:44.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d88650640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d8009dad0 con 0x7f2d8009ccc0 2024-09-17T15:45:44.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d88650640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:44.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d88650640 10 monclient(hunting): authenticate will time out at 755.766610s 2024-09-17T15:45:44.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d863ec640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f2d800a5600 0x7f2d800a59e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:41180/0 (socket says 172.21.15.100:41180) 2024-09-17T15:45:44.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d85beb640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f2d8009ccc0 0x7f2d8013e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:33922/0 (socket says 172.21.15.100:33922) 2024-09-17T15:45:44.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d863ec640 1 -- 172.21.15.100:0/931810878 learned_addr learned my addr 172.21.15.100:0/931810878 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-17T15:45:44.386 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d853ea640 1 -- 172.21.15.100:0/931810878 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2292462037 0 0) 0x7f2d8009dad0 con 0x7f2d8009ccc0 2024-09-17T15:45:44.386 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d853ea640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.386 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.362+0000 7f2d853ea640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient(hunting): my global_id is 4116 2024-09-17T15:45:44.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 1 -- 172.21.15.100:0/931810878 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d6c003520 con 0x7f2d8009ccc0 2024-09-17T15:45:44.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 1 -- 172.21.15.100:0/931810878 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129424021 0 0) 0x7f2d8009d7f0 con 0x7f2d800a5600 2024-09-17T15:45:44.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient(hunting): my global_id is 4141 2024-09-17T15:45:44.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 1 -- 172.21.15.100:0/931810878 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d8009dad0 con 0x7f2d800a5600 2024-09-17T15:45:44.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 1 -- 172.21.15.100:0/931810878 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2350482511 0 0) 0x7f2d80139140 con 0x7f2d8013f720 2024-09-17T15:45:44.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient(hunting): my global_id is 4104 2024-09-17T15:45:44.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 1 -- 172.21.15.100:0/931810878 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d8009d7f0 con 0x7f2d8013f720 2024-09-17T15:45:44.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 1 -- 172.21.15.100:0/931810878 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3976415357 0 0) 0x7f2d6c003520 con 0x7f2d8009ccc0 2024-09-17T15:45:44.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 1 -- 172.21.15.100:0/931810878 >> v1:172.21.15.100:6789/0 conn(0x7f2d800a5600 legacy=0x7f2d800a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 1 -- 172.21.15.100:0/931810878 >> v1:172.21.15.64:6789/0 conn(0x7f2d8013f720 legacy=0x7f2d80141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.389 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient: _finish_hunting 0 2024-09-17T15:45:44.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 1 monclient: found mon.noname-g 2024-09-17T15:45:44.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:44.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.100:6790/0 2024-09-17T15:45:44.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 1 -- 172.21.15.100:0/931810878 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8009ddb0 con 0x7f2d8009ccc0 2024-09-17T15:45:44.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient: _finish_auth 0 2024-09-17T15:45:44.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient: _check_auth_tickets 2024-09-17T15:45:44.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:44.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 1 -- 172.21.15.100:0/931810878 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2d7c001830 con 0x7f2d8009ccc0 2024-09-17T15:45:44.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:45:44.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient: dump: 2024-09-17T15:45:44.391 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.391 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.392 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.392 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.392 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.392 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.392 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.392 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.392 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.392 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.393 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.393 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.393 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.393 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.393 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.393 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d88650640 5 monclient: authenticate success, global_id 4116 2024-09-17T15:45:44.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d88650640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:44.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 1 -- 172.21.15.100:0/931810878 <== mon.3 v1:172.21.15.100:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d7c001a50 con 0x7f2d8009ccc0 2024-09-17T15:45:44.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:44.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d88650640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:44.393 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 1 -- 172.21.15.100:0/931810878 <== mon.3 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2d7c004b20 con 0x7f2d8009ccc0 2024-09-17T15:45:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d88650640 4 set_mon_vals no callback set 2024-09-17T15:45:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:45:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d853ea640 10 monclient: dump: 2024-09-17T15:45:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d88650640 10 monclient: shutdown 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d88650640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.363+0000 7f2d88650640 1 -- 172.21.15.100:0/931810878 >> v1:172.21.15.100:6790/0 conn(0x7f2d8009ccc0 legacy=0x7f2d8013e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.364+0000 7f2d88650640 1 -- 172.21.15.100:0/931810878 shutdown_connections 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.364+0000 7f2d88650640 1 -- 172.21.15.100:0/931810878 >> 172.21.15.100:0/931810878 conn(0x7f2d8006db40 msgr2=0x7f2d800aba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.364+0000 7f2d88650640 1 -- 172.21.15.100:0/931810878 shutdown_connections 2024-09-17T15:45:44.395 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.364+0000 7f2d88650640 1 -- 172.21.15.100:0/931810878 wait complete. 2024-09-17T15:45:44.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.364+0000 7f2d88650640 10 monclient: build_initial_monmap 2024-09-17T15:45:44.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.364+0000 7f2d88650640 10 monclient: monmap: 2024-09-17T15:45:44.396 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-17T15:45:44.396 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:44.396 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-17T15:45:44.396 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-17T15:45:44.396 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:44.396 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-17T15:45:44.396 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:44.396 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:44.396 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:44.396 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:44.397 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:44.397 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:44.397 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:44.397 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:44.397 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:44.397 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.364+0000 7f2d88650640 1 Processor -- start 2024-09-17T15:45:44.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d88650640 1 -- start start 2024-09-17T15:45:44.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d88650640 10 monclient: init 2024-09-17T15:45:44.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d88650640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:44.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d88650640 10 monclient: _add_conns ranks=[8,2,5,1,4,0,3,6,7] 2024-09-17T15:45:44.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d88650640 10 monclient(hunting): picked mon.noname-e con 0x7f2d8009ccc0 addr v1:172.21.15.64:6793/0 2024-09-17T15:45:44.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d88650640 10 monclient(hunting): picked mon.noname-b con 0x7f2d800a5600 addr v1:172.21.15.64:6790/0 2024-09-17T15:45:44.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d88650640 10 monclient(hunting): picked mon.noname-h con 0x7f2d8013f720 addr v1:172.21.15.100:6791/0 2024-09-17T15:45:44.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d88650640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d80134c40 con 0x7f2d800a5600 2024-09-17T15:45:44.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d88650640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d80134f20 con 0x7f2d8013f720 2024-09-17T15:45:44.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d88650640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d80135200 con 0x7f2d8009ccc0 2024-09-17T15:45:44.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d88650640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:44.398 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d88650640 10 monclient(hunting): authenticate will time out at 755.769590s 2024-09-17T15:45:44.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d86bed640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7f2d8013f720 0x7f2d801d63a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.100:47938/0 (socket says 172.21.15.100:47938) 2024-09-17T15:45:44.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d863ec640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f2d8009ccc0 0x7f2d801ccbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.100:34506/0 (socket says 172.21.15.100:34506) 2024-09-17T15:45:44.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d86bed640 1 -- 172.21.15.100:0/1923241046 learned_addr learned my addr 172.21.15.100:0/1923241046 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-17T15:45:44.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d85beb640 1 --1- 172.21.15.100:0/1923241046 >> v1:172.21.15.64:6790/0 conn(0x7f2d800a5600 0x7f2d801d2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.100:59538/0 (socket says 172.21.15.100:59538) 2024-09-17T15:45:44.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1986678826 0 0) 0x7f2d80134f20 con 0x7f2d8013f720 2024-09-17T15:45:44.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 10 monclient(hunting): my global_id is 4118 2024-09-17T15:45:44.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d64003550 con 0x7f2d8013f720 2024-09-17T15:45:44.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2511399757 0 0) 0x7f2d80135200 con 0x7f2d8009ccc0 2024-09-17T15:45:44.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.399 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 10 monclient(hunting): my global_id is 4121 2024-09-17T15:45:44.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d80134f20 con 0x7f2d8009ccc0 2024-09-17T15:45:44.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3500467578 0 0) 0x7f2d80134c40 con 0x7f2d800a5600 2024-09-17T15:45:44.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 10 monclient(hunting): my global_id is 4115 2024-09-17T15:45:44.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.365+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d80135200 con 0x7f2d800a5600 2024-09-17T15:45:44.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1128882158 0 0) 0x7f2d64003550 con 0x7f2d8013f720 2024-09-17T15:45:44.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d80134c40 con 0x7f2d8013f720 2024-09-17T15:45:44.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2d78001dd0 con 0x7f2d8013f720 2024-09-17T15:45:44.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:44.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-17T15:45:44.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient(hunting): dump: 2024-09-17T15:45:44.401 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.401 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.401 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.401 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.401 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.401 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.401 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.401 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.401 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.401 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.401 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.402 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.402 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.402 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.402 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.402 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:45:44.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3135350046 0 0) 0x7f2d80134f20 con 0x7f2d8009ccc0 2024-09-17T15:45:44.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d64003550 con 0x7f2d8009ccc0 2024-09-17T15:45:44.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2d70001470 con 0x7f2d8009ccc0 2024-09-17T15:45:44.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:44.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:45:44.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient(hunting): dump: 2024-09-17T15:45:44.402 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.403 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3057673038 0 0) 0x7f2d80135200 con 0x7f2d800a5600 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d80134f20 con 0x7f2d800a5600 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2d7c0017f0 con 0x7f2d800a5600 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient(hunting): dump: 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.404 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.405 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.405 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.405 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.405 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.405 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.405 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.405 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.405 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.405 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1020733795 0 0) 0x7f2d80134c40 con 0x7f2d8013f720 2024-09-17T15:45:44.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 >> v1:172.21.15.64:6793/0 conn(0x7f2d8009ccc0 legacy=0x7f2d801ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 >> v1:172.21.15.64:6790/0 conn(0x7f2d800a5600 legacy=0x7f2d801d2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 monclient: found mon.h 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 --> v1:172.21.15.100:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d80135a00 con 0x7f2d8013f720 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient: _finish_auth 0 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d88650640 5 monclient: authenticate success, global_id 4118 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d88650640 10 monclient: _renew_subs 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d88650640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d88650640 1 -- 172.21.15.100:0/1923241046 --> v1:172.21.15.100:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2d80132bc0 con 0x7f2d8013f720 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d88650640 10 monclient: _renew_subs 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d88650640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:45:44.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d88650640 1 -- 172.21.15.100:0/1923241046 --> v1:172.21.15.100:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f2d80133be0 con 0x7f2d8013f720 2024-09-17T15:45:44.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.5 v1:172.21.15.100:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d78002070 con 0x7f2d8013f720 2024-09-17T15:45:44.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d76ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:44.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.366+0000 7f2d84be9640 4 set_mon_vals no callback set 2024-09-17T15:45:44.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.367+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.5 v1:172.21.15.100:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2d78003f80 con 0x7f2d8013f720 2024-09-17T15:45:44.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.367+0000 7f2d76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.367+0000 7f2d76ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:45:44.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.367+0000 7f2d76ffd640 10 monclient: dump: 2024-09-17T15:45:44.407 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.407 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.407 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.407 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.408 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.408 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.408 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.408 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.408 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.408 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.408 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.408 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.408 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.409 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.409 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.409 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.367+0000 7f2d88650640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.367+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.5 v1:172.21.15.100:6791/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2d78004570 con 0x7f2d8013f720 2024-09-17T15:45:44.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.367+0000 7f2d88650640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.409 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.367+0000 7f2d88650640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:45:44.410 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.367+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.5 v1:172.21.15.100:6791/0 8 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (unknown 1129255178 0 0) 0x7f2d78006c10 con 0x7f2d8013f720 2024-09-17T15:45:44.410 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.367+0000 7f2d88650640 1 -- 172.21.15.100:0/1923241046 --> v1:172.21.15.100:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d80069560 con 0x7f2d8013f720 2024-09-17T15:45:44.410 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.369+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.5 v1:172.21.15.100:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f2d80069560 con 0x7f2d8013f720 2024-09-17T15:45:44.410 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.369+0000 7f2d76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.410 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.369+0000 7f2d76ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:44.441 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.439+0000 7f2d88650640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6b8dcb8b-604c-46d3-a682-5ef3e93155d0", "id": 1}] 2024-09-17T15:45:44.442 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.439+0000 7f2d88650640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6b8dcb8b-604c-46d3-a682-5ef3e93155d0", "id": 1}] 2024-09-17T15:45:44.442 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.439+0000 7f2d88650640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:45:44.442 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.439+0000 7f2d88650640 1 -- 172.21.15.100:0/1923241046 --> v1:172.21.15.100:6791/0 -- mon_command({"prefix": "osd new", "uuid": "6b8dcb8b-604c-46d3-a682-5ef3e93155d0", "id": 1} v 0) -- 0x7f2d8009dad0 con 0x7f2d8013f720 2024-09-17T15:45:44.444 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.442+0000 7f2d76ffd640 1 -- 172.21.15.100:0/1923241046 <== mon.5 v1:172.21.15.100:6791/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6b8dcb8b-604c-46d3-a682-5ef3e93155d0", "id": 1}]=0 v4) ==== 112+0+2 (unknown 1260994151 0 940483276) 0x7f2d8009dad0 con 0x7f2d8013f720 2024-09-17T15:45:44.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.442+0000 7f2d76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6b8dcb8b-604c-46d3-a682-5ef3e93155d0", "id": 1}] 2024-09-17T15:45:44.445 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.442+0000 7f2d76ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:44.445 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-09-17T15:45:44.446 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.443+0000 7f2d88650640 10 monclient: shutdown 2024-09-17T15:45:44.447 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.443+0000 7f2d88650640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:44.447 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.443+0000 7f2d88650640 1 -- 172.21.15.100:0/1923241046 >> v1:172.21.15.100:6791/0 conn(0x7f2d8013f720 legacy=0x7f2d801d63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.447 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.443+0000 7f2d88650640 1 -- 172.21.15.100:0/1923241046 shutdown_connections 2024-09-17T15:45:44.447 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.443+0000 7f2d88650640 1 -- 172.21.15.100:0/1923241046 >> 172.21.15.100:0/1923241046 conn(0x7f2d8006db40 msgr2=0x7f2d8012f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:44.447 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.444+0000 7f2d88650640 1 -- 172.21.15.100:0/1923241046 shutdown_connections 2024-09-17T15:45:44.447 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.444+0000 7f2d88650640 1 -- 172.21.15.100:0/1923241046 wait complete. 2024-09-17T15:45:44.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 34a266c9-43fe-4e7b-ae85-df0115c5f618 2 2024-09-17T15:45:44.551 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.548+0000 7fa117789640 10 monclient: get_monmap_and_config 2024-09-17T15:45:44.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.548+0000 7fa117789640 10 monclient: build_initial_monmap 2024-09-17T15:45:44.552 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.548+0000 7fa117789640 10 monclient: monmap: 2024-09-17T15:45:44.552 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-17T15:45:44.552 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.552 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:44.550259+0000 2024-09-17T15:45:44.552 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:44.550259+0000 2024-09-17T15:45:44.552 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:44.552 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.552 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:44.552 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:44.553 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.549+0000 7fa117789640 1 Processor -- start 2024-09-17T15:45:44.553 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.549+0000 7fa117789640 1 -- start start 2024-09-17T15:45:44.554 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.549+0000 7fa117789640 10 monclient: init 2024-09-17T15:45:44.561 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa117789640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:44.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa117789640 10 monclient: _add_conns ranks=[8,1,5,2,0,3,6,4,7] 2024-09-17T15:45:44.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa117789640 10 monclient(hunting): picked mon.noname-e con 0x7fa1100a5600 addr v1:172.21.15.64:6793/0 2024-09-17T15:45:44.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa117789640 10 monclient(hunting): picked mon.noname-f con 0x7fa11009ccc0 addr v1:172.21.15.100:6789/0 2024-09-17T15:45:44.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa117789640 10 monclient(hunting): picked mon.noname-h con 0x7fa11013f720 addr v1:172.21.15.100:6791/0 2024-09-17T15:45:44.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa117789640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa110139140 con 0x7fa11009ccc0 2024-09-17T15:45:44.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa117789640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa11009d7f0 con 0x7fa11013f720 2024-09-17T15:45:44.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa117789640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa11009dad0 con 0x7fa1100a5600 2024-09-17T15:45:44.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa117789640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:44.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa117789640 10 monclient(hunting): authenticate will time out at 755.954242s 2024-09-17T15:45:44.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa115d26640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7fa11013f720 0x7fa110141b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.100:47950/0 (socket says 172.21.15.100:47950) 2024-09-17T15:45:44.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa115d26640 1 -- 172.21.15.100:0/924275942 learned_addr learned my addr 172.21.15.100:0/924275942 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-17T15:45:44.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa114d24640 1 --1- 172.21.15.100:0/924275942 >> v1:172.21.15.100:6789/0 conn(0x7fa11009ccc0 0x7fa11013e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:41190/0 (socket says 172.21.15.100:41190) 2024-09-17T15:45:44.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 1 -- 172.21.15.100:0/924275942 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4042997609 0 0) 0x7fa110139140 con 0x7fa11009ccc0 2024-09-17T15:45:44.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient(hunting): my global_id is 4150 2024-09-17T15:45:44.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 1 -- 172.21.15.100:0/924275942 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0fc003520 con 0x7fa11009ccc0 2024-09-17T15:45:44.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 1 -- 172.21.15.100:0/924275942 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3183961861 0 0) 0x7fa11009d7f0 con 0x7fa11013f720 2024-09-17T15:45:44.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.563 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient(hunting): my global_id is 4127 2024-09-17T15:45:44.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 1 -- 172.21.15.100:0/924275942 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa110139140 con 0x7fa11013f720 2024-09-17T15:45:44.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 1 -- 172.21.15.100:0/924275942 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2972317248 0 0) 0x7fa110139140 con 0x7fa11013f720 2024-09-17T15:45:44.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 1 -- 172.21.15.100:0/924275942 >> v1:172.21.15.64:6793/0 conn(0x7fa1100a5600 legacy=0x7fa1100a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 1 -- 172.21.15.100:0/924275942 >> v1:172.21.15.100:6789/0 conn(0x7fa11009ccc0 legacy=0x7fa11013e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient: _finish_hunting 0 2024-09-17T15:45:44.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 1 monclient: found mon.noname-h 2024-09-17T15:45:44.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:44.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.100:6791/0 2024-09-17T15:45:44.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 1 -- 172.21.15.100:0/924275942 --> v1:172.21.15.100:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa11009ddb0 con 0x7fa11013f720 2024-09-17T15:45:44.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient: _finish_auth 0 2024-09-17T15:45:44.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient: _check_auth_tickets 2024-09-17T15:45:44.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:44.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa117789640 5 monclient: authenticate success, global_id 4127 2024-09-17T15:45:44.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 1 -- 172.21.15.100:0/924275942 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa108001b90 con 0x7fa11013f720 2024-09-17T15:45:44.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-17T15:45:44.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.550+0000 7fa103fff640 10 monclient: dump: 2024-09-17T15:45:44.565 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.566 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.566 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.566 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.566 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.566 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.566 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.566 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.566 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.566 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.566 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.567 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.567 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.567 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.567 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.567 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa117789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:44.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa103fff640 1 -- 172.21.15.100:0/924275942 <== mon.5 v1:172.21.15.100:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa108001d30 con 0x7fa11013f720 2024-09-17T15:45:44.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa103fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:44.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa117789640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:44.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa103fff640 1 -- 172.21.15.100:0/924275942 <== mon.5 v1:172.21.15.100:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa108004f60 con 0x7fa11013f720 2024-09-17T15:45:44.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa117789640 4 set_mon_vals no callback set 2024-09-17T15:45:44.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa103fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:45:44.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa103fff640 10 monclient: dump: 2024-09-17T15:45:44.568 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.568 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.568 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.568 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.568 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.568 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.568 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.568 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.568 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.569 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.569 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.569 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.569 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.569 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.569 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.569 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa117789640 10 monclient: shutdown 2024-09-17T15:45:44.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa117789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:44.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa117789640 1 -- 172.21.15.100:0/924275942 >> v1:172.21.15.100:6791/0 conn(0x7fa11013f720 legacy=0x7fa110141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa117789640 1 -- 172.21.15.100:0/924275942 shutdown_connections 2024-09-17T15:45:44.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa117789640 1 -- 172.21.15.100:0/924275942 >> 172.21.15.100:0/924275942 conn(0x7fa11006db40 msgr2=0x7fa1100aba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:44.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa117789640 1 -- 172.21.15.100:0/924275942 shutdown_connections 2024-09-17T15:45:44.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.551+0000 7fa117789640 1 -- 172.21.15.100:0/924275942 wait complete. 2024-09-17T15:45:44.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.552+0000 7fa117789640 10 monclient: build_initial_monmap 2024-09-17T15:45:44.570 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.552+0000 7fa117789640 10 monclient: monmap: 2024-09-17T15:45:44.570 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-17T15:45:44.570 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:44.570 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-17T15:45:44.570 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-17T15:45:44.570 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:44.570 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-17T15:45:44.570 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:44.571 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:44.571 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:44.571 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:44.571 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:44.571 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:44.571 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:44.571 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:44.571 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:44.571 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.552+0000 7fa117789640 1 Processor -- start 2024-09-17T15:45:44.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.552+0000 7fa117789640 1 -- start start 2024-09-17T15:45:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.552+0000 7fa117789640 10 monclient: init 2024-09-17T15:45:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa117789640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa117789640 10 monclient: _add_conns ranks=[0,7,4,5,1,8,3,6,2] 2024-09-17T15:45:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa117789640 10 monclient(hunting): picked mon.noname-a con 0x7fa11009ccc0 addr v1:172.21.15.64:6789/0 2024-09-17T15:45:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa117789640 10 monclient(hunting): picked mon.noname-i con 0x7fa1100a5600 addr v1:172.21.15.100:6792/0 2024-09-17T15:45:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa117789640 10 monclient(hunting): picked mon.noname-c con 0x7fa11013f720 addr v1:172.21.15.64:6791/0 2024-09-17T15:45:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa117789640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1100a0760 con 0x7fa11009ccc0 2024-09-17T15:45:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa117789640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa11009f3b0 con 0x7fa11013f720 2024-09-17T15:45:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa117789640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1100a51e0 con 0x7fa1100a5600 2024-09-17T15:45:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa117789640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa117789640 10 monclient(hunting): authenticate will time out at 755.957165s 2024-09-17T15:45:44.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa114d24640 1 --1- >> v1:172.21.15.100:6792/0 conn(0x7fa1100a5600 0x7fa11009eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6792/0 says I am v1:172.21.15.100:45480/0 (socket says 172.21.15.100:45480) 2024-09-17T15:45:44.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa114d24640 1 -- 172.21.15.100:0/277366102 learned_addr learned my addr 172.21.15.100:0/277366102 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-17T15:45:44.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2466437532 0 0) 0x7fa1100a51e0 con 0x7fa1100a5600 2024-09-17T15:45:44.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): my global_id is 4129 2024-09-17T15:45:44.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0f4003520 con 0x7fa1100a5600 2024-09-17T15:45:44.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1174621006 0 0) 0x7fa1100a0760 con 0x7fa11009ccc0 2024-09-17T15:45:44.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): my global_id is 4113 2024-09-17T15:45:44.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1100a51e0 con 0x7fa11009ccc0 2024-09-17T15:45:44.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2753677256 0 0) 0x7fa11009f3b0 con 0x7fa11013f720 2024-09-17T15:45:44.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): my global_id is 4126 2024-09-17T15:45:44.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1100a0760 con 0x7fa11013f720 2024-09-17T15:45:44.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4105856702 0 0) 0x7fa0f4003520 con 0x7fa1100a5600 2024-09-17T15:45:44.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa11009f3b0 con 0x7fa1100a5600 2024-09-17T15:45:44.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa104001650 con 0x7fa1100a5600 2024-09-17T15:45:44.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:44.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-17T15:45:44.575 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): dump: 2024-09-17T15:45:44.575 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.575 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.575 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.575 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.575 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.575 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.575 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.576 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.576 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.576 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.576 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.576 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.576 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.576 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.576 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.576 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:45:44.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3219132944 0 0) 0x7fa1100a51e0 con 0x7fa11009ccc0 2024-09-17T15:45:44.576 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0f4003520 con 0x7fa11009ccc0 2024-09-17T15:45:44.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa10c001a90 con 0x7fa11009ccc0 2024-09-17T15:45:44.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:44.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:44.577 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient(hunting): dump: 2024-09-17T15:45:44.577 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.577 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.577 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.577 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.577 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.577 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.577 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.578 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.578 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.578 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.578 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.578 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.578 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.578 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.579 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.579 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4219519936 0 0) 0x7fa11009f3b0 con 0x7fa1100a5600 2024-09-17T15:45:44.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 >> v1:172.21.15.64:6791/0 conn(0x7fa11013f720 legacy=0x7fa1101d5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 >> v1:172.21.15.64:6789/0 conn(0x7fa11009ccc0 legacy=0x7fa11009e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:45:44.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.553+0000 7fa101ffb640 1 monclient: found mon.i 2024-09-17T15:45:44.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:44.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:45:44.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 --> v1:172.21.15.100:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1100a0a40 con 0x7fa1100a5600 2024-09-17T15:45:44.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 10 monclient: _finish_auth 0 2024-09-17T15:45:44.579 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:45:44.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:44.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa117789640 5 monclient: authenticate success, global_id 4129 2024-09-17T15:45:44.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa117789640 10 monclient: _renew_subs 2024-09-17T15:45:44.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa117789640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:45:44.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa117789640 1 -- 172.21.15.100:0/277366102 --> v1:172.21.15.100:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa11013f3a0 con 0x7fa1100a5600 2024-09-17T15:45:44.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.7 v1:172.21.15.100:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa104004a90 con 0x7fa1100a5600 2024-09-17T15:45:44.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:44.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.7 v1:172.21.15.100:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa104004f80 con 0x7fa1100a5600 2024-09-17T15:45:44.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:45:44.580 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 10 monclient: dump: 2024-09-17T15:45:44.580 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.581 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.581 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.581 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.581 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.581 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.581 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.581 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.581 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.581 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.581 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.581 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.581 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.581 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.582 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.582 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa117789640 10 monclient: _renew_subs 2024-09-17T15:45:44.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa1037fe640 4 set_mon_vals no callback set 2024-09-17T15:45:44.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa117789640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:45:44.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa117789640 1 -- 172.21.15.100:0/277366102 --> v1:172.21.15.100:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fa11013ea80 con 0x7fa1100a5600 2024-09-17T15:45:44.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.7 v1:172.21.15.100:6792/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa1040054f0 con 0x7fa1100a5600 2024-09-17T15:45:44.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.554+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.7 v1:172.21.15.100:6792/0 8 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (unknown 1904504282 0 0) 0x7fa104005c00 con 0x7fa1100a5600 2024-09-17T15:45:44.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.555+0000 7fa117789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.555+0000 7fa117789640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.555+0000 7fa117789640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:45:44.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.555+0000 7fa117789640 1 -- 172.21.15.100:0/277366102 --> v1:172.21.15.100:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1100008d0 con 0x7fa1100a5600 2024-09-17T15:45:44.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.558+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.7 v1:172.21.15.100:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fa1040057e0 con 0x7fa1100a5600 2024-09-17T15:45:44.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.558+0000 7fa101ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.583 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.558+0000 7fa101ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:44.634 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.632+0000 7fa117789640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "34a266c9-43fe-4e7b-ae85-df0115c5f618", "id": 2}] 2024-09-17T15:45:44.635 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.632+0000 7fa117789640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "34a266c9-43fe-4e7b-ae85-df0115c5f618", "id": 2}] 2024-09-17T15:45:44.635 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.632+0000 7fa117789640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:45:44.635 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.632+0000 7fa117789640 1 -- 172.21.15.100:0/277366102 --> v1:172.21.15.100:6792/0 -- mon_command({"prefix": "osd new", "uuid": "34a266c9-43fe-4e7b-ae85-df0115c5f618", "id": 2} v 0) -- 0x7fa11013da10 con 0x7fa1100a5600 2024-09-17T15:45:44.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.641+0000 7fa101ffb640 1 -- 172.21.15.100:0/277366102 <== mon.7 v1:172.21.15.100:6792/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "34a266c9-43fe-4e7b-ae85-df0115c5f618", "id": 2}]=0 v5) ==== 112+0+2 (unknown 2766714932 0 216599125) 0x7fa104005e00 con 0x7fa1100a5600 2024-09-17T15:45:44.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.641+0000 7fa101ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "34a266c9-43fe-4e7b-ae85-df0115c5f618", "id": 2}] 2024-09-17T15:45:44.643 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.641+0000 7fa101ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:44.643 INFO:teuthology.orchestra.run.smithi100.stdout:2 2024-09-17T15:45:44.644 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.642+0000 7fa117789640 10 monclient: shutdown 2024-09-17T15:45:44.644 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.642+0000 7fa117789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:44.645 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.642+0000 7fa117789640 1 -- 172.21.15.100:0/277366102 >> v1:172.21.15.100:6792/0 conn(0x7fa1100a5600 legacy=0x7fa11009eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.645 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.642+0000 7fa117789640 1 -- 172.21.15.100:0/277366102 shutdown_connections 2024-09-17T15:45:44.645 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.642+0000 7fa117789640 1 -- 172.21.15.100:0/277366102 >> 172.21.15.100:0/277366102 conn(0x7fa11006db40 msgr2=0x7fa110137f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:44.645 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.642+0000 7fa117789640 1 -- 172.21.15.100:0/277366102 shutdown_connections 2024-09-17T15:45:44.645 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.642+0000 7fa117789640 1 -- 172.21.15.100:0/277366102 wait complete. 2024-09-17T15:45:44.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new c8740bab-d046-474f-b6a4-84333eb1e094 3 2024-09-17T15:45:44.750 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.748+0000 7fa2b18e2640 10 monclient: get_monmap_and_config 2024-09-17T15:45:44.751 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.748+0000 7fa2b18e2640 10 monclient: build_initial_monmap 2024-09-17T15:45:44.751 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.748+0000 7fa2b18e2640 10 monclient: monmap: 2024-09-17T15:45:44.751 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-17T15:45:44.751 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.751 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:44.750099+0000 2024-09-17T15:45:44.751 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:44.750099+0000 2024-09-17T15:45:44.751 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:44.751 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.751 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:44.752 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:44.752 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:44.752 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:44.752 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:44.752 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:44.752 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:44.752 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:44.752 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:44.752 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.749+0000 7fa2b18e2640 1 Processor -- start 2024-09-17T15:45:44.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.749+0000 7fa2b18e2640 1 -- start start 2024-09-17T15:45:44.759 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.749+0000 7fa2b18e2640 10 monclient: init 2024-09-17T15:45:44.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.749+0000 7fa2b18e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:44.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.749+0000 7fa2b18e2640 10 monclient: _add_conns ranks=[5,1,3,7,8,6,0,2,4] 2024-09-17T15:45:44.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.749+0000 7fa2b18e2640 10 monclient(hunting): picked mon.noname-h con 0x7fa2ac0a5600 addr v1:172.21.15.100:6791/0 2024-09-17T15:45:44.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2b18e2640 10 monclient(hunting): picked mon.noname-f con 0x7fa2ac09ccc0 addr v1:172.21.15.100:6789/0 2024-09-17T15:45:44.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2b18e2640 10 monclient(hunting): picked mon.noname-g con 0x7fa2ac13f720 addr v1:172.21.15.100:6790/0 2024-09-17T15:45:44.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2b18e2640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2ac139140 con 0x7fa2ac09ccc0 2024-09-17T15:45:44.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2b18e2640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2ac09d7f0 con 0x7fa2ac13f720 2024-09-17T15:45:44.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2b18e2640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2ac09dad0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2b18e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:44.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2b18e2640 10 monclient(hunting): authenticate will time out at 756.154097s 2024-09-17T15:45:44.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aaffd640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7fa2ac0a5600 0x7fa2ac0a59e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.100:47958/0 (socket says 172.21.15.100:47958) 2024-09-17T15:45:44.760 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aaffd640 1 -- 172.21.15.100:0/2579567064 learned_addr learned my addr 172.21.15.100:0/2579567064 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 -- 172.21.15.100:0/2579567064 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 213909030 0 0) 0x7fa2ac09dad0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient(hunting): my global_id is 4136 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 -- 172.21.15.100:0/2579567064 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa298003520 con 0x7fa2ac0a5600 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 -- 172.21.15.100:0/2579567064 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1533251578 0 0) 0x7fa2ac139140 con 0x7fa2ac09ccc0 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient(hunting): my global_id is 4159 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 -- 172.21.15.100:0/2579567064 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2ac09dad0 con 0x7fa2ac09ccc0 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 -- 172.21.15.100:0/2579567064 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2956559130 0 0) 0x7fa2ac09d7f0 con 0x7fa2ac13f720 2024-09-17T15:45:44.761 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient(hunting): my global_id is 4125 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 -- 172.21.15.100:0/2579567064 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2ac139140 con 0x7fa2ac13f720 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 -- 172.21.15.100:0/2579567064 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1279260094 0 0) 0x7fa298003520 con 0x7fa2ac0a5600 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 -- 172.21.15.100:0/2579567064 >> v1:172.21.15.100:6790/0 conn(0x7fa2ac13f720 legacy=0x7fa2ac141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 -- 172.21.15.100:0/2579567064 >> v1:172.21.15.100:6789/0 conn(0x7fa2ac09ccc0 legacy=0x7fa2ac13e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 monclient: found mon.noname-h 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.100:6791/0 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 -- 172.21.15.100:0/2579567064 --> v1:172.21.15.100:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ac09ddb0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient: _finish_auth 0 2024-09-17T15:45:44.762 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 -- 172.21.15.100:0/2579567064 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa29c001830 con 0x7fa2ac0a5600 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2b18e2640 5 monclient: authenticate success, global_id 4136 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2b18e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient: dump: 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.764 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.764 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.764 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.764 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.764 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.764 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.764 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.764 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.764 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2b18e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:44.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 -- 172.21.15.100:0/2579567064 <== mon.5 v1:172.21.15.100:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa29c001a50 con 0x7fa2ac0a5600 2024-09-17T15:45:44.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:44.764 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 1 -- 172.21.15.100:0/2579567064 <== mon.5 v1:172.21.15.100:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa29c004be0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2b18e2640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2b18e2640 4 set_mon_vals no callback set 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.750+0000 7fa2aa7fc640 10 monclient: dump: 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.765 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.751+0000 7fa2b18e2640 10 monclient: shutdown 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.751+0000 7fa2b18e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.751+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2579567064 >> v1:172.21.15.100:6791/0 conn(0x7fa2ac0a5600 legacy=0x7fa2ac0a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.751+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2579567064 shutdown_connections 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.751+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2579567064 >> 172.21.15.100:0/2579567064 conn(0x7fa2ac06db40 msgr2=0x7fa2ac0aba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.751+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2579567064 shutdown_connections 2024-09-17T15:45:44.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.751+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2579567064 wait complete. 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.752+0000 7fa2b18e2640 10 monclient: build_initial_monmap 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.752+0000 7fa2b18e2640 10 monclient: monmap: 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:44.767 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.752+0000 7fa2b18e2640 1 Processor -- start 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.752+0000 7fa2b18e2640 1 -- start start 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.752+0000 7fa2b18e2640 10 monclient: init 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.752+0000 7fa2b18e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 10 monclient: _add_conns ranks=[8,1,6,4,0,7,5,3,2] 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 10 monclient(hunting): picked mon.noname-e con 0x7fa2ac09ccc0 addr v1:172.21.15.64:6793/0 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 10 monclient(hunting): picked mon.noname-f con 0x7fa2ac0a5600 addr v1:172.21.15.100:6789/0 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 10 monclient(hunting): picked mon.noname-d con 0x7fa2ac13f720 addr v1:172.21.15.64:6792/0 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2ac0a0780 con 0x7fa2ac0a5600 2024-09-17T15:45:44.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2ac13f1e0 con 0x7fa2ac13f720 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2ac13e4e0 con 0x7fa2ac09ccc0 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 10 monclient(hunting): authenticate will time out at 756.157113s 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a3fff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fa2ac0a5600 0x7fa2ac1d2ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:41216/0 (socket says 172.21.15.100:41216) 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a3fff640 1 -- 172.21.15.100:0/2160809332 learned_addr learned my addr 172.21.15.100:0/2160809332 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3788707320 0 0) 0x7fa2ac09dad0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): my global_id is 4168 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa28c003550 con 0x7fa2ac0a5600 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3668420467 0 0) 0x7fa28c003550 con 0x7fa2ac0a5600 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa2ac09dad0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa294001af0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): dump: 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.770 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2085799600 0 0) 0x7fa2ac13e4e0 con 0x7fa2ac09ccc0 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): my global_id is 4139 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa28c003550 con 0x7fa2ac09ccc0 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3989159589 0 0) 0x7fa2ac13f1e0 con 0x7fa2ac13f720 2024-09-17T15:45:44.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient(hunting): my global_id is 4128 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2ac13e4e0 con 0x7fa2ac13f720 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 928203520 0 0) 0x7fa2ac09dad0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 >> v1:172.21.15.64:6793/0 conn(0x7fa2ac09ccc0 legacy=0x7fa2ac1326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 >> v1:172.21.15.64:6792/0 conn(0x7fa2ac13f720 legacy=0x7fa2ac1d6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient: _finish_hunting 0 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 monclient: found mon.f 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ac0a1020 con 0x7fa2ac0a5600 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient: _finish_auth 0 2024-09-17T15:45:44.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:45:44.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:44.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 5 monclient: authenticate success, global_id 4168 2024-09-17T15:45:44.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 10 monclient: _renew_subs 2024-09-17T15:45:44.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:45:44.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2160809332 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa2ac13ebc0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 10 monclient: _renew_subs 2024-09-17T15:45:44.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 <== mon.1 v1:172.21.15.100:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa2940038c0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:45:44.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2160809332 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa2ac13ddc0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:44.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 <== mon.1 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa294003e30 con 0x7fa2ac0a5600 2024-09-17T15:45:44.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:45:44.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a8ff9640 10 monclient: dump: 2024-09-17T15:45:44.774 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.774 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.774 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.774 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.774 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.774 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.774 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.774 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.774 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.774 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.774 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.774 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.754+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 <== mon.1 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa2940057c0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.753+0000 7fa2a9ffb640 4 set_mon_vals no callback set 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.754+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 <== mon.1 v1:172.21.15.100:6789/0 8 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (unknown 2198298400 0 0) 0x7fa294006060 con 0x7fa2ac0a5600 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.754+0000 7fa2b18e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.754+0000 7fa2b18e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.754+0000 7fa2b18e2640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.754+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2160809332 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2ac0008d0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.756+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 <== mon.1 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fa2ac0a0780 con 0x7fa2ac0a5600 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.756+0000 7fa2a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.756+0000 7fa2a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:44.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.827+0000 7fa2b18e2640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c8740bab-d046-474f-b6a4-84333eb1e094", "id": 3}] 2024-09-17T15:45:44.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.827+0000 7fa2b18e2640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c8740bab-d046-474f-b6a4-84333eb1e094", "id": 3}] 2024-09-17T15:45:44.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.827+0000 7fa2b18e2640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:45:44.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.827+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2160809332 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c8740bab-d046-474f-b6a4-84333eb1e094", "id": 3} v 0) -- 0x7fa2ac132dd0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.836+0000 7fa2a8ff9640 1 -- 172.21.15.100:0/2160809332 <== mon.1 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c8740bab-d046-474f-b6a4-84333eb1e094", "id": 3}]=0 v6) ==== 112+0+2 (unknown 1423832764 0 525046306) 0x7fa2940043e0 con 0x7fa2ac0a5600 2024-09-17T15:45:44.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.836+0000 7fa2a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c8740bab-d046-474f-b6a4-84333eb1e094", "id": 3}] 2024-09-17T15:45:44.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.836+0000 7fa2a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:44.838 INFO:teuthology.orchestra.run.smithi100.stdout:3 2024-09-17T15:45:44.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.837+0000 7fa2b18e2640 10 monclient: shutdown 2024-09-17T15:45:44.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.837+0000 7fa2b18e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:44.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.837+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2160809332 >> v1:172.21.15.100:6789/0 conn(0x7fa2ac0a5600 legacy=0x7fa2ac1d2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.837+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2160809332 shutdown_connections 2024-09-17T15:45:44.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.837+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2160809332 >> 172.21.15.100:0/2160809332 conn(0x7fa2ac06db40 msgr2=0x7fa2ac1374a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:44.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.837+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2160809332 shutdown_connections 2024-09-17T15:45:44.840 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.837+0000 7fa2b18e2640 1 -- 172.21.15.100:0/2160809332 wait complete. 2024-09-17T15:45:44.851 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 08a1700b-63c5-4028-824f-dbd38931ddb8 4 2024-09-17T15:45:44.947 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.944+0000 7f03f7efe640 10 monclient: get_monmap_and_config 2024-09-17T15:45:44.947 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.944+0000 7f03f7efe640 10 monclient: build_initial_monmap 2024-09-17T15:45:44.947 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.944+0000 7f03f7efe640 10 monclient: monmap: 2024-09-17T15:45:44.947 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-17T15:45:44.947 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.947 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:44.946178+0000 2024-09-17T15:45:44.947 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:44.946178+0000 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.945+0000 7f03f7efe640 1 Processor -- start 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.945+0000 7f03f7efe640 1 -- start start 2024-09-17T15:45:44.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.945+0000 7f03f7efe640 10 monclient: init 2024-09-17T15:45:44.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.945+0000 7f03f7efe640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:44.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 10 monclient: _add_conns ranks=[8,0,5,4,1,3,7,6,2] 2024-09-17T15:45:44.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 10 monclient(hunting): picked mon.noname-e con 0x7f03f00a5600 addr v1:172.21.15.64:6793/0 2024-09-17T15:45:44.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 10 monclient(hunting): picked mon.noname-a con 0x7f03f009ccc0 addr v1:172.21.15.64:6789/0 2024-09-17T15:45:44.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 10 monclient(hunting): picked mon.noname-h con 0x7f03f013f720 addr v1:172.21.15.100:6791/0 2024-09-17T15:45:44.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f0139140 con 0x7f03f009ccc0 2024-09-17T15:45:44.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f009d7f0 con 0x7f03f013f720 2024-09-17T15:45:44.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f009dad0 con 0x7f03f00a5600 2024-09-17T15:45:44.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:44.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 10 monclient(hunting): authenticate will time out at 756.350123s 2024-09-17T15:45:44.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f649b640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7f03f013f720 0x7f03f0141b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.100:47964/0 (socket says 172.21.15.100:47964) 2024-09-17T15:45:44.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f649b640 1 -- 172.21.15.100:0/3117435021 learned_addr learned my addr 172.21.15.100:0/3117435021 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-17T15:45:44.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 -- 172.21.15.100:0/3117435021 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1135224591 0 0) 0x7f03f009d7f0 con 0x7f03f013f720 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient(hunting): my global_id is 4145 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 -- 172.21.15.100:0/3117435021 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03dc003520 con 0x7f03f013f720 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 -- 172.21.15.100:0/3117435021 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1075762607 0 0) 0x7f03f0139140 con 0x7f03f009ccc0 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient(hunting): my global_id is 4122 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 -- 172.21.15.100:0/3117435021 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03f009d7f0 con 0x7f03f009ccc0 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 -- 172.21.15.100:0/3117435021 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3011601749 0 0) 0x7f03f009dad0 con 0x7f03f00a5600 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient(hunting): my global_id is 4148 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 -- 172.21.15.100:0/3117435021 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03f0139140 con 0x7f03f00a5600 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 -- 172.21.15.100:0/3117435021 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1651626612 0 0) 0x7f03dc003520 con 0x7f03f013f720 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 -- 172.21.15.100:0/3117435021 >> v1:172.21.15.64:6793/0 conn(0x7f03f00a5600 legacy=0x7f03f00a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 -- 172.21.15.100:0/3117435021 >> v1:172.21.15.64:6789/0 conn(0x7f03f009ccc0 legacy=0x7f03f013e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient: _finish_hunting 0 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 monclient: found mon.noname-h 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.100:6791/0 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 -- 172.21.15.100:0/3117435021 --> v1:172.21.15.100:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f009ddb0 con 0x7f03f013f720 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient: _finish_auth 0 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient: _check_auth_tickets 2024-09-17T15:45:44.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:44.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 5 monclient: authenticate success, global_id 4145 2024-09-17T15:45:44.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 -- 172.21.15.100:0/3117435021 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f03e8001bb0 con 0x7f03f013f720 2024-09-17T15:45:44.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:44.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-17T15:45:44.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient: dump: 2024-09-17T15:45:44.960 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.960 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.960 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.960 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.960 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.960 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.960 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 -- 172.21.15.100:0/3117435021 <== mon.5 v1:172.21.15.100:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f03e8001dd0 con 0x7f03f013f720 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 1 -- 172.21.15.100:0/3117435021 <== mon.5 v1:172.21.15.100:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f03e8004f60 con 0x7f03f013f720 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:44.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f7efe640 4 set_mon_vals no callback set 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.946+0000 7f03f4c98640 10 monclient: dump: 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.962 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.947+0000 7f03f7efe640 10 monclient: shutdown 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.947+0000 7f03f7efe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.947+0000 7f03f7efe640 1 -- 172.21.15.100:0/3117435021 >> v1:172.21.15.100:6791/0 conn(0x7f03f013f720 legacy=0x7f03f0141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.947+0000 7f03f7efe640 1 -- 172.21.15.100:0/3117435021 shutdown_connections 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.947+0000 7f03f7efe640 1 -- 172.21.15.100:0/3117435021 >> 172.21.15.100:0/3117435021 conn(0x7f03f006db40 msgr2=0x7f03f00aba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.947+0000 7f03f7efe640 1 -- 172.21.15.100:0/3117435021 shutdown_connections 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.947+0000 7f03f7efe640 1 -- 172.21.15.100:0/3117435021 wait complete. 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.948+0000 7f03f7efe640 10 monclient: build_initial_monmap 2024-09-17T15:45:44.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.948+0000 7f03f7efe640 10 monclient: monmap: 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:44.964 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:44.965 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.948+0000 7f03f7efe640 1 Processor -- start 2024-09-17T15:45:44.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.948+0000 7f03f7efe640 1 -- start start 2024-09-17T15:45:44.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.948+0000 7f03f7efe640 10 monclient: init 2024-09-17T15:45:44.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.948+0000 7f03f7efe640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:44.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.948+0000 7f03f7efe640 10 monclient: _add_conns ranks=[4,2,6,8,3,7,0,1,5] 2024-09-17T15:45:44.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.948+0000 7f03f7efe640 10 monclient(hunting): picked mon.noname-c con 0x7f03f009ccc0 addr v1:172.21.15.64:6791/0 2024-09-17T15:45:44.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03f7efe640 10 monclient(hunting): picked mon.noname-b con 0x7f03f00a5600 addr v1:172.21.15.64:6790/0 2024-09-17T15:45:44.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03f7efe640 10 monclient(hunting): picked mon.noname-d con 0x7f03f013f720 addr v1:172.21.15.64:6792/0 2024-09-17T15:45:44.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03f7efe640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f00a1290 con 0x7f03f00a5600 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03f7efe640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f009ecd0 con 0x7f03f009ccc0 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03f7efe640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f013e6b0 con 0x7f03f013f720 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03f7efe640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03f7efe640 10 monclient(hunting): authenticate will time out at 756.353090s 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03f5c9a640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f03f009ccc0 0x7f03f00a47c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.100:45958/0 (socket says 172.21.15.100:45958) 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03f5c9a640 1 -- 172.21.15.100:0/2572619288 learned_addr learned my addr 172.21.15.100:0/2572619288 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1966651494 0 0) 0x7f03f009ecd0 con 0x7f03f009ccc0 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): my global_id is 4135 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03d4003550 con 0x7f03f009ccc0 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1653710359 0 0) 0x7f03f00a1290 con 0x7f03f00a5600 2024-09-17T15:45:44.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): my global_id is 4124 2024-09-17T15:45:44.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03f009ecd0 con 0x7f03f00a5600 2024-09-17T15:45:44.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3842986252 0 0) 0x7f03f013e6b0 con 0x7f03f013f720 2024-09-17T15:45:44.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:44.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:44.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:44.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): my global_id is 4137 2024-09-17T15:45:44.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03f00a1290 con 0x7f03f013f720 2024-09-17T15:45:44.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 936887526 0 0) 0x7f03d4003550 con 0x7f03f009ccc0 2024-09-17T15:45:44.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03f013e6b0 con 0x7f03f009ccc0 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f03ec001b20 con 0x7f03f009ccc0 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): dump: 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.968 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.969 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.969 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.969 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.969 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.969 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.969 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:45:44.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3866851742 0 0) 0x7f03f009ecd0 con 0x7f03f00a5600 2024-09-17T15:45:44.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03d4003550 con 0x7f03f00a5600 2024-09-17T15:45:44.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f03e4001ad0 con 0x7f03f00a5600 2024-09-17T15:45:44.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:44.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:44.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): dump: 2024-09-17T15:45:44.970 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.970 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.970 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.970 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.970 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.970 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.970 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.970 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.970 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.970 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3635800210 0 0) 0x7f03f00a1290 con 0x7f03f013f720 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03f009ecd0 con 0x7f03f013f720 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f03e8001360 con 0x7f03f013f720 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.949+0000 7f03e27fc640 10 monclient(hunting): dump: 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.971 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.972 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.972 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.972 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.972 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.972 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.972 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.972 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.972 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.972 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.972 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.972 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.972 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.972 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3971922643 0 0) 0x7f03f013e6b0 con 0x7f03f009ccc0 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 >> v1:172.21.15.64:6792/0 conn(0x7f03f013f720 legacy=0x7f03f01d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 >> v1:172.21.15.64:6790/0 conn(0x7f03f00a5600 legacy=0x7f03f01d2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 10 monclient: _finish_hunting 0 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 1 monclient: found mon.c 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f00a09f0 con 0x7f03f009ccc0 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 10 monclient: _finish_auth 0 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03f7efe640 5 monclient: authenticate success, global_id 4135 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03f7efe640 10 monclient: _renew_subs 2024-09-17T15:45:44.973 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03f7efe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03f7efe640 1 -- 172.21.15.100:0/2572619288 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f03f009fa90 con 0x7f03f009ccc0 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03f7efe640 10 monclient: _renew_subs 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03f7efe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03f7efe640 1 -- 172.21.15.100:0/2572619288 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f03f00a50b0 con 0x7f03f009ccc0 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f03ec0038c0 con 0x7f03f009ccc0 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f03ec003df0 con 0x7f03f009ccc0 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e3fff640 4 set_mon_vals no callback set 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 10 monclient: dump: 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.975 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:44.975 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:44.975 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:44.975 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:44.975 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:44.975 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:44.975 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:44.975 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:44.975 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:44.975 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:44.976 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:44.976 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:44.976 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:44.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f03ec0057c0 con 0x7f03f009ccc0 2024-09-17T15:45:44.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.950+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (unknown 3095815639 0 0) 0x7f03ec006ca0 con 0x7f03f009ccc0 2024-09-17T15:45:44.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.951+0000 7f03f7efe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.951+0000 7f03f7efe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.951+0000 7f03f7efe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:44.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.951+0000 7f03f7efe640 1 -- 172.21.15.100:0/2572619288 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03d8005080 con 0x7f03f009ccc0 2024-09-17T15:45:44.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.953+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f03ec005c90 con 0x7f03f009ccc0 2024-09-17T15:45:44.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.953+0000 7f03e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:44.976 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:44.953+0000 7f03e27fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:45.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.022+0000 7f03f7efe640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "08a1700b-63c5-4028-824f-dbd38931ddb8", "id": 4}] 2024-09-17T15:45:45.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.022+0000 7f03f7efe640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "08a1700b-63c5-4028-824f-dbd38931ddb8", "id": 4}] 2024-09-17T15:45:45.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.022+0000 7f03f7efe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:45.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.022+0000 7f03f7efe640 1 -- 172.21.15.100:0/2572619288 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "osd new", "uuid": "08a1700b-63c5-4028-824f-dbd38931ddb8", "id": 4} v 0) -- 0x7f03d8005370 con 0x7f03f009ccc0 2024-09-17T15:45:45.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.026+0000 7f03e27fc640 1 -- 172.21.15.100:0/2572619288 <== mon.4 v1:172.21.15.64:6791/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "08a1700b-63c5-4028-824f-dbd38931ddb8", "id": 4}]=0 v7) ==== 112+0+2 (unknown 3091466002 0 1697012583) 0x7f03ec005fc0 con 0x7f03f009ccc0 2024-09-17T15:45:45.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.026+0000 7f03e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "08a1700b-63c5-4028-824f-dbd38931ddb8", "id": 4}] 2024-09-17T15:45:45.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.026+0000 7f03e27fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:45.029 INFO:teuthology.orchestra.run.smithi100.stdout:4 2024-09-17T15:45:45.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.027+0000 7f03f7efe640 10 monclient: shutdown 2024-09-17T15:45:45.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.027+0000 7f03f7efe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:45.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.027+0000 7f03f7efe640 1 -- 172.21.15.100:0/2572619288 >> v1:172.21.15.64:6791/0 conn(0x7f03f009ccc0 legacy=0x7f03f00a47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:45.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.027+0000 7f03f7efe640 1 -- 172.21.15.100:0/2572619288 shutdown_connections 2024-09-17T15:45:45.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.027+0000 7f03f7efe640 1 -- 172.21.15.100:0/2572619288 >> 172.21.15.100:0/2572619288 conn(0x7f03f006db40 msgr2=0x7f03f0137590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:45.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.027+0000 7f03f7efe640 1 -- 172.21.15.100:0/2572619288 shutdown_connections 2024-09-17T15:45:45.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.027+0000 7f03f7efe640 1 -- 172.21.15.100:0/2572619288 wait complete. 2024-09-17T15:45:45.042 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 81fa403f-39e9-44fd-8b1f-ef72fd7ba855 5 2024-09-17T15:45:45.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.136+0000 7f4961d9b640 10 monclient: get_monmap_and_config 2024-09-17T15:45:45.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.136+0000 7f4961d9b640 10 monclient: build_initial_monmap 2024-09-17T15:45:45.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.136+0000 7f4961d9b640 10 monclient: monmap: 2024-09-17T15:45:45.139 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-17T15:45:45.139 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.139 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:45.138312+0000 2024-09-17T15:45:45.139 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:45.138312+0000 2024-09-17T15:45:45.139 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:45.139 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:45.140 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:45.140 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:45.140 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:45.140 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:45.140 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:45.140 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:45.140 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:45.140 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:45.140 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:45.140 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:45.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.137+0000 7f4961d9b640 1 Processor -- start 2024-09-17T15:45:45.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.137+0000 7f4961d9b640 1 -- start start 2024-09-17T15:45:45.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.137+0000 7f4961d9b640 10 monclient: init 2024-09-17T15:45:45.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.138+0000 7f4961d9b640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:45.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.138+0000 7f4961d9b640 10 monclient: _add_conns ranks=[6,1,5,8,3,0,4,7,2] 2024-09-17T15:45:45.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.138+0000 7f4961d9b640 10 monclient(hunting): picked mon.noname-d con 0x7f495c0a5bf0 addr v1:172.21.15.64:6792/0 2024-09-17T15:45:45.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.138+0000 7f4961d9b640 10 monclient(hunting): picked mon.noname-f con 0x7f495c0a2830 addr v1:172.21.15.100:6789/0 2024-09-17T15:45:45.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.138+0000 7f4961d9b640 10 monclient(hunting): picked mon.noname-h con 0x7f495c13f790 addr v1:172.21.15.100:6791/0 2024-09-17T15:45:45.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.138+0000 7f4961d9b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f495c0a5fd0 con 0x7f495c0a2830 2024-09-17T15:45:45.148 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.138+0000 7f4961d9b640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f495c0a4e00 con 0x7f495c13f790 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.138+0000 7f4961d9b640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f495c0693c0 con 0x7f495c0a5bf0 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.138+0000 7f4961d9b640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.138+0000 7f4961d9b640 10 monclient(hunting): authenticate will time out at 756.542761s 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.138+0000 7f495affd640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f495c0a2830 0x7f495c0a46f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:41220/0 (socket says 172.21.15.100:41220) 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.138+0000 7f495affd640 1 -- 172.21.15.100:0/1601086811 learned_addr learned my addr 172.21.15.100:0/1601086811 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.138+0000 7f495bfff640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7f495c13f790 0x7f495c141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.100:47972/0 (socket says 172.21.15.100:47972) 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 1 -- 172.21.15.100:0/1601086811 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 476961878 0 0) 0x7f495c0a5fd0 con 0x7f495c0a2830 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient(hunting): my global_id is 4177 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 1 -- 172.21.15.100:0/1601086811 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4944003520 con 0x7f495c0a2830 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 1 -- 172.21.15.100:0/1601086811 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3599395534 0 0) 0x7f495c0a4e00 con 0x7f495c13f790 2024-09-17T15:45:45.149 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient(hunting): my global_id is 4154 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 1 -- 172.21.15.100:0/1601086811 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f495c0a5fd0 con 0x7f495c13f790 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 1 -- 172.21.15.100:0/1601086811 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1026292957 0 0) 0x7f4944003520 con 0x7f495c0a2830 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 1 -- 172.21.15.100:0/1601086811 >> v1:172.21.15.64:6792/0 conn(0x7f495c0a5bf0 legacy=0x7f495c0a3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 1 -- 172.21.15.100:0/1601086811 >> v1:172.21.15.100:6791/0 conn(0x7f495c13f790 legacy=0x7f495c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 1 monclient: found mon.noname-f 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.100:6789/0 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 1 -- 172.21.15.100:0/1601086811 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495c0a05b0 con 0x7f495c0a2830 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:45:45.150 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 1 -- 172.21.15.100:0/1601086811 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4950001830 con 0x7f495c0a2830 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f4961d9b640 5 monclient: authenticate success, global_id 4177 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f4961d9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient: dump: 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:45.151 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f4961d9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 1 -- 172.21.15.100:0/1601086811 <== mon.1 v1:172.21.15.100:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4950001a50 con 0x7f495c0a2830 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 1 -- 172.21.15.100:0/1601086811 <== mon.1 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4950001d40 con 0x7f495c0a2830 2024-09-17T15:45:45.152 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f4961d9b640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f4961d9b640 4 set_mon_vals no callback set 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.139+0000 7f495a7fc640 10 monclient: dump: 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:45.153 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.140+0000 7f4961d9b640 10 monclient: shutdown 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.140+0000 7f4961d9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.140+0000 7f4961d9b640 1 -- 172.21.15.100:0/1601086811 >> v1:172.21.15.100:6789/0 conn(0x7f495c0a2830 legacy=0x7f495c0a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.140+0000 7f4961d9b640 1 -- 172.21.15.100:0/1601086811 shutdown_connections 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.140+0000 7f4961d9b640 1 -- 172.21.15.100:0/1601086811 >> 172.21.15.100:0/1601086811 conn(0x7f495c0a50d0 msgr2=0x7f495c0a54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.140+0000 7f4961d9b640 1 -- 172.21.15.100:0/1601086811 shutdown_connections 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.140+0000 7f4961d9b640 1 -- 172.21.15.100:0/1601086811 wait complete. 2024-09-17T15:45:45.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.141+0000 7f4961d9b640 10 monclient: build_initial_monmap 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.141+0000 7f4961d9b640 10 monclient: monmap: 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:45.155 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.141+0000 7f4961d9b640 1 Processor -- start 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.141+0000 7f4961d9b640 1 -- start start 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.141+0000 7f4961d9b640 10 monclient: init 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4961d9b640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4961d9b640 10 monclient: _add_conns ranks=[0,3,7,6,8,4,2,1,5] 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4961d9b640 10 monclient(hunting): picked mon.noname-a con 0x7f495c0a5bf0 addr v1:172.21.15.64:6789/0 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4961d9b640 10 monclient(hunting): picked mon.noname-g con 0x7f495c13f790 addr v1:172.21.15.100:6790/0 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4961d9b640 10 monclient(hunting): picked mon.noname-i con 0x7f495c1d3f30 addr v1:172.21.15.100:6792/0 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4961d9b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f495c1ce810 con 0x7f495c0a5bf0 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4961d9b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f495c13bac0 con 0x7f495c13f790 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4961d9b640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f495c0a6a00 con 0x7f495c1d3f30 2024-09-17T15:45:45.156 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4961d9b640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4961d9b640 10 monclient(hunting): authenticate will time out at 756.546524s 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f495affd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f495c13f790 0x7f495c1ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:33936/0 (socket says 172.21.15.100:33936) 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f495affd640 1 -- 172.21.15.100:0/2955398626 learned_addr learned my addr 172.21.15.100:0/2955398626 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3141131129 0 0) 0x7f495c13bac0 con 0x7f495c13f790 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 10 monclient(hunting): _negotiate 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 10 monclient(hunting): my global_id is 4134 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4948003520 con 0x7f495c13f790 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3575423625 0 0) 0x7f495c0a6a00 con 0x7f495c1d3f30 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 10 monclient(hunting): _negotiate 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:45.157 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 10 monclient(hunting): my global_id is 4138 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f495c13bac0 con 0x7f495c1d3f30 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2223833375 0 0) 0x7f495c1ce810 con 0x7f495c0a5bf0 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 10 monclient(hunting): _negotiate 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 10 monclient(hunting): my global_id is 4131 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.142+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f495c0a6a00 con 0x7f495c0a5bf0 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1477885301 0 0) 0x7f4948003520 con 0x7f495c13f790 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f495c1ce810 con 0x7f495c13f790 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f49500017f0 con 0x7f495c13f790 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient(hunting): dump: 2024-09-17T15:45:45.158 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 515139067 0 0) 0x7f495c13bac0 con 0x7f495c1d3f30 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4948003520 con 0x7f495c1d3f30 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4954001b40 con 0x7f495c1d3f30 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient(hunting): dump: 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:45.160 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 833912666 0 0) 0x7f495c0a6a00 con 0x7f495c0a5bf0 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f495c13bac0 con 0x7f495c0a5bf0 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f494c001a10 con 0x7f495c0a5bf0 2024-09-17T15:45:45.161 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:45.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:45.162 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient(hunting): dump: 2024-09-17T15:45:45.162 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:45.162 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.162 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.162 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.162 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:45.162 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:45.162 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:45.162 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:45.162 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:45.162 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:45.162 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4158952282 0 0) 0x7f495c1ce810 con 0x7f495c13f790 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 >> v1:172.21.15.100:6792/0 conn(0x7f495c1d3f30 legacy=0x7f495c1d6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 >> v1:172.21.15.64:6789/0 conn(0x7f495c0a5bf0 legacy=0x7f495c1cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient: _finish_hunting 0 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 monclient: found mon.g 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495c09f130 con 0x7f495c13f790 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient: _finish_auth 0 2024-09-17T15:45:45.163 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient: _check_auth_tickets 2024-09-17T15:45:45.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:45.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4961d9b640 5 monclient: authenticate success, global_id 4134 2024-09-17T15:45:45.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4961d9b640 10 monclient: _renew_subs 2024-09-17T15:45:45.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4961d9b640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:45:45.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4961d9b640 1 -- 172.21.15.100:0/2955398626 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f495c073e50 con 0x7f495c13f790 2024-09-17T15:45:45.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4961d9b640 10 monclient: _renew_subs 2024-09-17T15:45:45.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4961d9b640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:45:45.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4961d9b640 1 -- 172.21.15.100:0/2955398626 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f495c09eb20 con 0x7f495c13f790 2024-09-17T15:45:45.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.3 v1:172.21.15.100:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4950001cc0 con 0x7f495c13f790 2024-09-17T15:45:45.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:45.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.3 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4950005270 con 0x7f495c13f790 2024-09-17T15:45:45.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:45.164 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4959ffb640 4 set_mon_vals no callback set 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 10 monclient: dump: 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 3 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:45.165 INFO:teuthology.orchestra.run.smithi100.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:45.166 INFO:teuthology.orchestra.run.smithi100.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:45.166 INFO:teuthology.orchestra.run.smithi100.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:45.166 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:45:45.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.3 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f49500057e0 con 0x7f495c13f790 2024-09-17T15:45:45.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.143+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.3 v1:172.21.15.100:6790/0 8 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (unknown 4047210365 0 0) 0x7f4950006c40 con 0x7f495c13f790 2024-09-17T15:45:45.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.144+0000 7f4961d9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:45.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.144+0000 7f4961d9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:45.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.144+0000 7f4961d9b640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:45:45.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.144+0000 7f4961d9b640 1 -- 172.21.15.100:0/2955398626 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f493c005080 con 0x7f495c13f790 2024-09-17T15:45:45.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.146+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.3 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f4950006770 con 0x7f495c13f790 2024-09-17T15:45:45.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.146+0000 7f4943fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:45.166 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.146+0000 7f4943fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:45.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.216+0000 7f4961d9b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "81fa403f-39e9-44fd-8b1f-ef72fd7ba855", "id": 5}] 2024-09-17T15:45:45.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.216+0000 7f4961d9b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "81fa403f-39e9-44fd-8b1f-ef72fd7ba855", "id": 5}] 2024-09-17T15:45:45.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.216+0000 7f4961d9b640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:45:45.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.216+0000 7f4961d9b640 1 -- 172.21.15.100:0/2955398626 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd new", "uuid": "81fa403f-39e9-44fd-8b1f-ef72fd7ba855", "id": 5} v 0) -- 0x7f493c005370 con 0x7f495c13f790 2024-09-17T15:45:45.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.221+0000 7f4943fff640 1 -- 172.21.15.100:0/2955398626 <== mon.3 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "81fa403f-39e9-44fd-8b1f-ef72fd7ba855", "id": 5}]=0 v8) ==== 112+0+2 (unknown 2450702171 0 1988412176) 0x7f4950005ad0 con 0x7f495c13f790 2024-09-17T15:45:45.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.221+0000 7f4943fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "81fa403f-39e9-44fd-8b1f-ef72fd7ba855", "id": 5}] 2024-09-17T15:45:45.223 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.221+0000 7f4943fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:45.223 INFO:teuthology.orchestra.run.smithi100.stdout:5 2024-09-17T15:45:45.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.222+0000 7f4961d9b640 10 monclient: shutdown 2024-09-17T15:45:45.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.222+0000 7f4961d9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:45.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.222+0000 7f4961d9b640 1 -- 172.21.15.100:0/2955398626 >> v1:172.21.15.100:6790/0 conn(0x7f495c13f790 legacy=0x7f495c1ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:45.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.222+0000 7f4961d9b640 1 -- 172.21.15.100:0/2955398626 shutdown_connections 2024-09-17T15:45:45.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.222+0000 7f4961d9b640 1 -- 172.21.15.100:0/2955398626 >> 172.21.15.100:0/2955398626 conn(0x7f495c0a50d0 msgr2=0x7f495c0a7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:45.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.222+0000 7f4961d9b640 1 -- 172.21.15.100:0/2955398626 shutdown_connections 2024-09-17T15:45:45.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-17T15:45:45.222+0000 7f4961d9b640 1 -- 172.21.15.100:0/2955398626 wait complete. 2024-09-17T15:45:45.234 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-17T15:45:45.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-17T15:45:45.237 INFO:tasks.ceph.osd.0:Started 2024-09-17T15:45:45.237 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-17T15:45:45.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-17T15:45:45.239 INFO:tasks.ceph.osd.1:Started 2024-09-17T15:45:45.239 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-17T15:45:45.240 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-17T15:45:45.242 INFO:tasks.ceph.osd.2:Started 2024-09-17T15:45:45.242 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-17T15:45:45.242 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-17T15:45:45.244 INFO:tasks.ceph.osd.3:Started 2024-09-17T15:45:45.244 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-17T15:45:45.244 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-17T15:45:45.246 INFO:tasks.ceph.osd.4:Started 2024-09-17T15:45:45.246 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-17T15:45:45.246 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-17T15:45:45.248 INFO:tasks.ceph.osd.5:Started 2024-09-17T15:45:45.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T15:45:45.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.364+0000 7fbd89305640 10 monclient: get_monmap_and_config 2024-09-17T15:45:45.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.364+0000 7fbd89305640 10 monclient: build_initial_monmap 2024-09-17T15:45:45.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.364+0000 7fbd89305640 10 monclient: monmap: 2024-09-17T15:45:45.367 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:45:45.367 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.367 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:45.365555+0000 2024-09-17T15:45:45.367 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:45.365555+0000 2024-09-17T15:45:45.367 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:45.367 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:45.367 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:45.368 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:45.368 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:45.368 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:45.368 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:45.368 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:45.368 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:45.369 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:45.369 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:45.369 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:45.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.365+0000 7fbd89305640 1 Processor -- start 2024-09-17T15:45:45.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.365+0000 7fbd89305640 1 -- start start 2024-09-17T15:45:45.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.365+0000 7fbd89305640 10 monclient: init 2024-09-17T15:45:45.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd89305640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:45.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd89305640 10 monclient: _add_conns ranks=[3,1,8,2,6,5,4,7,0] 2024-09-17T15:45:45.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd89305640 10 monclient(hunting): picked mon.noname-g con 0x7fbd840a54b0 addr v1:172.21.15.100:6790/0 2024-09-17T15:45:45.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd89305640 10 monclient(hunting): picked mon.noname-f con 0x7fbd84135c80 addr v1:172.21.15.100:6789/0 2024-09-17T15:45:45.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd89305640 10 monclient(hunting): picked mon.noname-e con 0x7fbd841360a0 addr v1:172.21.15.64:6793/0 2024-09-17T15:45:45.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd89305640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd840aacb0 con 0x7fbd84135c80 2024-09-17T15:45:45.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd89305640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd8409d780 con 0x7fbd840a54b0 2024-09-17T15:45:45.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd89305640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd8409da60 con 0x7fbd841360a0 2024-09-17T15:45:45.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd89305640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:45.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd89305640 10 monclient(hunting): authenticate will time out at 743.041313s 2024-09-17T15:45:45.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd837fe640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fbd841360a0 0x7fbd84141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:40112/0 (socket says 172.21.15.64:40112) 2024-09-17T15:45:45.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd837fe640 1 -- 172.21.15.64:0/2259764313 learned_addr learned my addr 172.21.15.64:0/2259764313 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:45:45.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 1 -- 172.21.15.64:0/2259764313 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3255578835 0 0) 0x7fbd8409da60 con 0x7fbd841360a0 2024-09-17T15:45:45.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:45.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:45.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:45.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient(hunting): my global_id is 4175 2024-09-17T15:45:45.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 1 -- 172.21.15.64:0/2259764313 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd70003520 con 0x7fbd841360a0 2024-09-17T15:45:45.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 1 -- 172.21.15.64:0/2259764313 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3148278153 0 0) 0x7fbd8409d780 con 0x7fbd840a54b0 2024-09-17T15:45:45.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:45.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:45.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:45.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient(hunting): my global_id is 4170 2024-09-17T15:45:45.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 1 -- 172.21.15.64:0/2259764313 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd8409da60 con 0x7fbd840a54b0 2024-09-17T15:45:45.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 1 -- 172.21.15.64:0/2259764313 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4230252549 0 0) 0x7fbd840aacb0 con 0x7fbd84135c80 2024-09-17T15:45:45.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:45.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient(hunting): my global_id is 4186 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 1 -- 172.21.15.64:0/2259764313 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd8409d780 con 0x7fbd84135c80 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 1 -- 172.21.15.64:0/2259764313 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4227084391 0 0) 0x7fbd8409da60 con 0x7fbd840a54b0 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 1 -- 172.21.15.64:0/2259764313 >> v1:172.21.15.64:6793/0 conn(0x7fbd841360a0 legacy=0x7fbd84141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 1 -- 172.21.15.64:0/2259764313 >> v1:172.21.15.100:6789/0 conn(0x7fbd84135c80 legacy=0x7fbd8413df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient: _finish_hunting 0 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 1 monclient: found mon.noname-g 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.100:6790/0 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 1 -- 172.21.15.64:0/2259764313 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8409dd40 con 0x7fbd840a54b0 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient: _finish_auth 0 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient: _check_auth_tickets 2024-09-17T15:45:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 1 -- 172.21.15.64:0/2259764313 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbd74001810 con 0x7fbd840a54b0 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd89305640 5 monclient: authenticate success, global_id 4170 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd89305640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd827fc640 10 monclient: dump: 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:45.386 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:45.387 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:45.387 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:45.387 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:45.387 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:45.387 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:45.387 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:45.387 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:45.387 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:45.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.366+0000 7fbd89305640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:45.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd827fc640 1 -- 172.21.15.64:0/2259764313 <== mon.3 v1:172.21.15.100:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd74001a30 con 0x7fbd840a54b0 2024-09-17T15:45:45.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd827fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:45.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd827fc640 1 -- 172.21.15.64:0/2259764313 <== mon.3 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbd74003950 con 0x7fbd840a54b0 2024-09-17T15:45:45.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd89305640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:45.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd89305640 4 set_mon_vals no callback set 2024-09-17T15:45:45.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:45.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd827fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:45:45.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd827fc640 10 monclient: dump: 2024-09-17T15:45:45.388 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:45:45.388 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.388 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.388 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.388 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:45.388 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:45.388 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:45.389 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:45.389 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:45.389 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:45.389 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:45.389 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:45.389 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:45.389 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:45.389 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:45.389 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:45.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd89305640 10 monclient: shutdown 2024-09-17T15:45:45.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd89305640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:45.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd89305640 1 -- 172.21.15.64:0/2259764313 >> v1:172.21.15.100:6790/0 conn(0x7fbd840a54b0 legacy=0x7fbd840a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:45.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd89305640 1 -- 172.21.15.64:0/2259764313 shutdown_connections 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd89305640 1 -- 172.21.15.64:0/2259764313 >> 172.21.15.64:0/2259764313 conn(0x7fbd8406daa0 msgr2=0x7fbd840ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd89305640 1 -- 172.21.15.64:0/2259764313 shutdown_connections 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.367+0000 7fbd89305640 1 -- 172.21.15.64:0/2259764313 wait complete. 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.368+0000 7fbd89305640 10 monclient: build_initial_monmap 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.368+0000 7fbd89305640 10 monclient: monmap: 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:45.390 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.368+0000 7fbd89305640 1 Processor -- start 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.368+0000 7fbd89305640 1 -- start start 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.368+0000 7fbd89305640 10 monclient: init 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd89305640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd89305640 10 monclient: _add_conns ranks=[6,5,4,7,3,1,2,0,8] 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd89305640 10 monclient(hunting): picked mon.noname-d con 0x7fbd840a54b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:45:45.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd89305640 10 monclient(hunting): picked mon.noname-h con 0x7fbd84135c80 addr v1:172.21.15.100:6791/0 2024-09-17T15:45:45.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd89305640 10 monclient(hunting): picked mon.noname-c con 0x7fbd841360a0 addr v1:172.21.15.64:6791/0 2024-09-17T15:45:45.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd89305640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd840a3950 con 0x7fbd841360a0 2024-09-17T15:45:45.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd89305640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd841cebf0 con 0x7fbd84135c80 2024-09-17T15:45:45.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd89305640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd840a3c30 con 0x7fbd840a54b0 2024-09-17T15:45:45.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd89305640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:45.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd89305640 10 monclient(hunting): authenticate will time out at 743.044130s 2024-09-17T15:45:45.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd82ffd640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fbd840a54b0 0x7fbd841cddd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:60966/0 (socket says 172.21.15.64:60966) 2024-09-17T15:45:45.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd82ffd640 1 -- 172.21.15.64:0/2521326200 learned_addr learned my addr 172.21.15.64:0/2521326200 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:45:45.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1270195136 0 0) 0x7fbd840a3950 con 0x7fbd841360a0 2024-09-17T15:45:45.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:45:45.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:45.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:45.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): my global_id is 4171 2024-09-17T15:45:45.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd64003520 con 0x7fbd841360a0 2024-09-17T15:45:45.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2823633999 0 0) 0x7fbd840a3c30 con 0x7fbd840a54b0 2024-09-17T15:45:45.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:45:45.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:45.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:45.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): my global_id is 4164 2024-09-17T15:45:45.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd840a3950 con 0x7fbd840a54b0 2024-09-17T15:45:45.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1913475675 0 0) 0x7fbd841cebf0 con 0x7fbd84135c80 2024-09-17T15:45:45.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:45:45.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:45.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:45.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): my global_id is 4199 2024-09-17T15:45:45.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd840a3c30 con 0x7fbd84135c80 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1438160485 0 0) 0x7fbd840a3c30 con 0x7fbd84135c80 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd841cebf0 con 0x7fbd84135c80 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbd6c0012d0 con 0x7fbd84135c80 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): dump: 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:45.394 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2122533882 0 0) 0x7fbd840a3950 con 0x7fbd840a54b0 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd840a3c30 con 0x7fbd840a54b0 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbd740017d0 con 0x7fbd840a54b0 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:45:45.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): dump: 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:45.397 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 41106836 0 0) 0x7fbd64003520 con 0x7fbd841360a0 2024-09-17T15:45:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd840a3950 con 0x7fbd841360a0 2024-09-17T15:45:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbd7c001c30 con 0x7fbd841360a0 2024-09-17T15:45:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 10 monclient(hunting): dump: 2024-09-17T15:45:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:45:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 778705673 0 0) 0x7fbd841cebf0 con 0x7fbd84135c80 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.369+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 >> v1:172.21.15.64:6792/0 conn(0x7fbd840a54b0 legacy=0x7fbd841cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:45.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 >> v1:172.21.15.64:6791/0 conn(0x7fbd841360a0 legacy=0x7fbd841d6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 1 monclient: found mon.h 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 --> v1:172.21.15.100:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd840a6ce0 con 0x7fbd84135c80 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd89305640 5 monclient: authenticate success, global_id 4199 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd89305640 10 monclient: _renew_subs 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd89305640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd89305640 1 -- 172.21.15.64:0/2521326200 --> v1:172.21.15.100:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbd840a6720 con 0x7fbd84135c80 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd89305640 10 monclient: _renew_subs 2024-09-17T15:45:45.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd89305640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:45:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd89305640 1 -- 172.21.15.64:0/2521326200 --> v1:172.21.15.100:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fbd840a4a50 con 0x7fbd84135c80 2024-09-17T15:45:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.5 v1:172.21.15.100:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd6c0035f0 con 0x7fbd84135c80 2024-09-17T15:45:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.5 v1:172.21.15.100:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbd6c003ae0 con 0x7fbd84135c80 2024-09-17T15:45:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:45:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 10 monclient: dump: 2024-09-17T15:45:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:45:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:45.400 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd81ffb640 4 set_mon_vals no callback set 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.5 v1:172.21.15.100:6791/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbd6c004010 con 0x7fbd84135c80 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.5 v1:172.21.15.100:6791/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 692025209 0 0) 0x7fbd6c005820 con 0x7fbd84135c80 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.370+0000 7fbd89305640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:45.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.371+0000 7fbd89305640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.371+0000 7fbd89305640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:45:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.371+0000 7fbd89305640 1 -- 172.21.15.64:0/2521326200 --> v1:172.21.15.100:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd8409d0d0 con 0x7fbd84135c80 2024-09-17T15:45:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.373+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.5 v1:172.21.15.100:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fbd8409d0d0 con 0x7fbd84135c80 2024-09-17T15:45:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.373+0000 7fbd7b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:45.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.373+0000 7fbd7b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:45.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.435+0000 7fbd89305640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:45:45.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.435+0000 7fbd89305640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:45:45.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.435+0000 7fbd89305640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:45:45.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.435+0000 7fbd89305640 1 -- 172.21.15.64:0/2521326200 --> v1:172.21.15.100:6791/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd840694b0 con 0x7fbd84135c80 2024-09-17T15:45:45.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.436+0000 7fbd7b7fe640 1 -- 172.21.15.64:0/2521326200 <== mon.5 v1:172.21.15.100:6791/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 2110007902) 0x7fbd840694b0 con 0x7fbd84135c80 2024-09-17T15:45:45.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.436+0000 7fbd7b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:45:45.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.436+0000 7fbd7b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:45.438 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:45:45.438 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":8,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","created":"2024-09-17T15:45:43.591977+0000","modified":"2024-09-17T15:45:45.218655+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T15:45:45.218655+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d9b48f9-b24c-453f-a5e9-79d74f56118c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6b8dcb8b-604c-46d3-a682-5ef3e93155d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34a266c9-43fe-4e7b-ae85-df0115c5f618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c8740bab-d046-474f-b6a4-84333eb1e094","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"08a1700b-63c5-4028-824f-dbd38931ddb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"81fa403f-39e9-44fd-8b1f-ef72fd7ba855","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:45:45.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.436+0000 7fbd89305640 10 monclient: shutdown 2024-09-17T15:45:45.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.436+0000 7fbd89305640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:45.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.436+0000 7fbd89305640 1 -- 172.21.15.64:0/2521326200 >> v1:172.21.15.100:6791/0 conn(0x7fbd84135c80 legacy=0x7fbd841ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:45.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.437+0000 7fbd89305640 1 -- 172.21.15.64:0/2521326200 shutdown_connections 2024-09-17T15:45:45.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.437+0000 7fbd89305640 1 -- 172.21.15.64:0/2521326200 >> 172.21.15.64:0/2521326200 conn(0x7fbd8406daa0 msgr2=0x7fbd840a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:45.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.437+0000 7fbd89305640 1 -- 172.21.15.64:0/2521326200 shutdown_connections 2024-09-17T15:45:45.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:45:45.437+0000 7fbd89305640 1 -- 172.21.15.64:0/2521326200 wait complete. 2024-09-17T15:45:45.448 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-17T15:45:45.448 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-17T15:45:46.053 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T15:45:46.150 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.149+0000 7f5943059640 10 monclient: get_monmap_and_config 2024-09-17T15:45:46.151 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.149+0000 7f5943059640 10 monclient: build_initial_monmap 2024-09-17T15:45:46.151 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.149+0000 7f5943059640 10 monclient: monmap: 2024-09-17T15:45:46.151 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-09-17T15:45:46.151 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:46.151 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:46.149954+0000 2024-09-17T15:45:46.151 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:46.149954+0000 2024-09-17T15:45:46.151 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:46.151 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:46.152 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:46.152 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:46.152 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:46.152 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:46.152 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:46.152 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:46.152 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:46.152 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:46.153 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:46.153 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:46.160 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.149+0000 7f5943059640 1 Processor -- start 2024-09-17T15:45:46.161 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.149+0000 7f5943059640 1 -- start start 2024-09-17T15:45:46.161 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.149+0000 7f5943059640 10 monclient: init 2024-09-17T15:45:46.161 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f5943059640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:46.161 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f5943059640 10 monclient: _add_conns ranks=[0,2,7,1,4,6,5,8,3] 2024-09-17T15:45:46.161 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f5943059640 10 monclient(hunting): picked mon.noname-a con 0x7f593c0a2510 addr v1:172.21.15.64:6789/0 2024-09-17T15:45:46.161 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f5943059640 10 monclient(hunting): picked mon.noname-b con 0x7f593c0a3000 addr v1:172.21.15.64:6790/0 2024-09-17T15:45:46.162 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f5943059640 10 monclient(hunting): picked mon.noname-i con 0x7f593c1439e0 addr v1:172.21.15.100:6792/0 2024-09-17T15:45:46.162 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f5943059640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f593c0a3ac0 con 0x7f593c0a2510 2024-09-17T15:45:46.162 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f5943059640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f593c0a1e60 con 0x7f593c0a3000 2024-09-17T15:45:46.162 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f5943059640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f593c09cb60 con 0x7f593c1439e0 2024-09-17T15:45:46.162 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f5943059640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:46.162 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f5943059640 10 monclient(hunting): authenticate will time out at 743.825449s 2024-09-17T15:45:46.163 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f5940df5640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f593c0a2510 0x7f593c0a28f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:38620/0 (socket says 172.21.15.64:38620) 2024-09-17T15:45:46.163 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593bfff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f593c0a3000 0x7f593c1422c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56784/0 (socket says 172.21.15.64:56784) 2024-09-17T15:45:46.163 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f5940df5640 1 -- 172.21.15.64:0/1957118267 learned_addr learned my addr 172.21.15.64:0/1957118267 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:45:46.163 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 1 -- 172.21.15.64:0/1957118267 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2639505543 0 0) 0x7f5928001040 con 0x7f593c0a3000 2024-09-17T15:45:46.163 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:45:46.163 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:46.164 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:46.164 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 10 monclient(hunting): my global_id is 4142 2024-09-17T15:45:46.164 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 1 -- 172.21.15.64:0/1957118267 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f592c003520 con 0x7f593c0a3000 2024-09-17T15:45:46.164 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 1 -- 172.21.15.64:0/1957118267 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3153711249 0 0) 0x7f593c0a3ac0 con 0x7f593c0a2510 2024-09-17T15:45:46.164 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:45:46.164 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:46.165 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:46.165 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 10 monclient(hunting): my global_id is 4149 2024-09-17T15:45:46.165 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 1 -- 172.21.15.64:0/1957118267 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5928001040 con 0x7f593c0a2510 2024-09-17T15:45:46.165 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 1 -- 172.21.15.64:0/1957118267 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2961476152 0 0) 0x7f593c09cb60 con 0x7f593c1439e0 2024-09-17T15:45:46.165 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:45:46.165 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:46.166 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:46.166 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 10 monclient(hunting): my global_id is 4174 2024-09-17T15:45:46.166 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.150+0000 7f593b7fe640 1 -- 172.21.15.64:0/1957118267 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f593c0a3ac0 con 0x7f593c1439e0 2024-09-17T15:45:46.166 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 1 -- 172.21.15.64:0/1957118267 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1041960583 0 0) 0x7f5928001040 con 0x7f593c0a2510 2024-09-17T15:45:46.166 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 1 -- 172.21.15.64:0/1957118267 >> v1:172.21.15.100:6792/0 conn(0x7f593c1439e0 legacy=0x7f593c145dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:46.167 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 1 -- 172.21.15.64:0/1957118267 >> v1:172.21.15.64:6790/0 conn(0x7f593c0a3000 legacy=0x7f593c1422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:46.167 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:45:46.167 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 1 monclient: found mon.noname-a 2024-09-17T15:45:46.167 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:46.167 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:45:46.167 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 1 -- 172.21.15.64:0/1957118267 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593c0a3cc0 con 0x7f593c0a2510 2024-09-17T15:45:46.168 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:45:46.168 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:45:46.168 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:46.168 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 1 -- 172.21.15.64:0/1957118267 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5930001810 con 0x7f593c0a2510 2024-09-17T15:45:46.168 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f5943059640 5 monclient: authenticate success, global_id 4149 2024-09-17T15:45:46.168 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f5943059640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:46.169 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:46.169 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:45:46.169 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 10 monclient: dump: 2024-09-17T15:45:46.169 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:46.169 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:46.169 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:46.170 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:46.170 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:46.170 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:46.170 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:46.170 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:46.171 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:46.171 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:46.171 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:46.171 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:46.171 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:46.171 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:46.172 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:46.172 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:46.172 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f5943059640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:46.172 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 1 -- 172.21.15.64:0/1957118267 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59300019b0 con 0x7f593c0a2510 2024-09-17T15:45:46.172 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:46.172 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 1 -- 172.21.15.64:0/1957118267 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5930004be0 con 0x7f593c0a2510 2024-09-17T15:45:46.173 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f5943059640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:46.173 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f5943059640 4 set_mon_vals no callback set 2024-09-17T15:45:46.173 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:46.173 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:46.173 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f593b7fe640 10 monclient: dump: 2024-09-17T15:45:46.173 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:46.174 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:46.174 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:46.174 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:46.174 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:46.174 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:46.174 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:46.175 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:46.175 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:46.175 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:46.175 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:46.175 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:46.176 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:46.176 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:46.176 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:46.176 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:46.176 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f5943059640 10 monclient: shutdown 2024-09-17T15:45:46.176 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f5943059640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:46.177 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f5943059640 1 -- 172.21.15.64:0/1957118267 >> v1:172.21.15.64:6789/0 conn(0x7f593c0a2510 legacy=0x7f593c0a28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:46.177 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f5943059640 1 -- 172.21.15.64:0/1957118267 shutdown_connections 2024-09-17T15:45:46.177 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f5943059640 1 -- 172.21.15.64:0/1957118267 >> 172.21.15.64:0/1957118267 conn(0x7f593c0a7f60 msgr2=0x7f593c0a8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:46.177 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f5943059640 1 -- 172.21.15.64:0/1957118267 shutdown_connections 2024-09-17T15:45:46.177 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.151+0000 7f5943059640 1 -- 172.21.15.64:0/1957118267 wait complete. 2024-09-17T15:45:46.177 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.152+0000 7f5943059640 10 monclient: build_initial_monmap 2024-09-17T15:45:46.178 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.152+0000 7f5943059640 10 monclient: monmap: 2024-09-17T15:45:46.178 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-09-17T15:45:46.178 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:46.178 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-09-17T15:45:46.178 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-09-17T15:45:46.178 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:46.179 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-09-17T15:45:46.179 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:46.179 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:46.179 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:46.180 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:46.180 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:46.180 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:46.180 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:46.180 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:46.181 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:46.181 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:46.181 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.152+0000 7f5943059640 1 Processor -- start 2024-09-17T15:45:46.181 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.152+0000 7f5943059640 1 -- start start 2024-09-17T15:45:46.181 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.152+0000 7f5943059640 10 monclient: init 2024-09-17T15:45:46.181 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f5943059640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:46.182 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f5943059640 10 monclient: _add_conns ranks=[3,4,2,8,5,7,6,0,1] 2024-09-17T15:45:46.182 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f5943059640 10 monclient(hunting): picked mon.noname-g con 0x7f593c0a2510 addr v1:172.21.15.100:6790/0 2024-09-17T15:45:46.182 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f5943059640 10 monclient(hunting): picked mon.noname-c con 0x7f593c0a3000 addr v1:172.21.15.64:6791/0 2024-09-17T15:45:46.182 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f5943059640 10 monclient(hunting): picked mon.noname-b con 0x7f593c1439e0 addr v1:172.21.15.64:6790/0 2024-09-17T15:45:46.182 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f5943059640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f593c12c4a0 con 0x7f593c1439e0 2024-09-17T15:45:46.182 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f5943059640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f593c1437b0 con 0x7f593c0a2510 2024-09-17T15:45:46.183 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f5943059640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f593c12d8f0 con 0x7f593c0a3000 2024-09-17T15:45:46.183 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f5943059640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:46.183 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f5943059640 10 monclient(hunting): authenticate will time out at 743.828688s 2024-09-17T15:45:46.183 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f59415f6640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f593c1439e0 0x7f593c1da620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56786/0 (socket says 172.21.15.64:56786) 2024-09-17T15:45:46.183 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f59415f6640 1 -- 172.21.15.64:0/427584231 learned_addr learned my addr 172.21.15.64:0/427584231 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:45:46.183 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2881450187 0 0) 0x7f593c12c4a0 con 0x7f593c1439e0 2024-09-17T15:45:46.184 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f59397fa640 10 monclient(hunting): _negotiate 2024-09-17T15:45:46.184 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f59397fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:46.184 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f59397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:46.184 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f59397fa640 10 monclient(hunting): my global_id is 4151 2024-09-17T15:45:46.184 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5920003520 con 0x7f593c1439e0 2024-09-17T15:45:46.185 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2359566741 0 0) 0x7f593c12d8f0 con 0x7f593c0a3000 2024-09-17T15:45:46.185 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f59397fa640 10 monclient(hunting): _negotiate 2024-09-17T15:45:46.185 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f59397fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:46.185 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f59397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:46.185 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.153+0000 7f59397fa640 10 monclient(hunting): my global_id is 4180 2024-09-17T15:45:46.185 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f593c12c4a0 con 0x7f593c0a3000 2024-09-17T15:45:46.186 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 341109944 0 0) 0x7f593c0a3cc0 con 0x7f593c0a2510 2024-09-17T15:45:46.186 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): _negotiate 2024-09-17T15:45:46.186 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:46.186 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:46.186 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): my global_id is 4179 2024-09-17T15:45:46.186 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f593c12d8f0 con 0x7f593c0a2510 2024-09-17T15:45:46.186 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3077451230 0 0) 0x7f5920003520 con 0x7f593c1439e0 2024-09-17T15:45:46.186 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f593c0a3cc0 con 0x7f593c1439e0 2024-09-17T15:45:46.187 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5934001db0 con 0x7f593c1439e0 2024-09-17T15:45:46.187 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:46.187 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:45:46.187 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): dump: 2024-09-17T15:45:46.187 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:46.187 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:46.188 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:46.188 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:46.188 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:46.188 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:46.188 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:46.188 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:46.188 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:46.188 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:46.188 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:46.188 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:46.188 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:46.189 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:46.189 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:46.189 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:46.189 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:45:46.189 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 425916368 0 0) 0x7f593c12c4a0 con 0x7f593c0a3000 2024-09-17T15:45:46.189 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5920003520 con 0x7f593c0a3000 2024-09-17T15:45:46.189 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2111890596 0 0) 0x7f593c12d8f0 con 0x7f593c0a2510 2024-09-17T15:45:46.189 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f593c12c4a0 con 0x7f593c0a2510 2024-09-17T15:45:46.189 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f59280016b0 con 0x7f593c0a3000 2024-09-17T15:45:46.189 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:46.189 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:46.190 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): dump: 2024-09-17T15:45:46.190 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:46.190 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:46.190 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:46.190 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:46.190 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:46.190 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:46.190 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:46.190 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:46.190 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:46.190 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:46.190 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:46.190 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:46.191 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:46.191 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:46.191 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:46.191 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:46.191 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5930004b60 con 0x7f593c0a2510 2024-09-17T15:45:46.191 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:46.191 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:45:46.191 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient(hunting): dump: 2024-09-17T15:45:46.191 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:46.191 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:46.191 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:46.191 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:46.192 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:46.192 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:46.192 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:46.192 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:46.192 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:46.192 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:46.192 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:46.192 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:46.192 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:46.192 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:46.192 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:46.192 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:46.192 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1200471253 0 0) 0x7f593c0a3cc0 con 0x7f593c1439e0 2024-09-17T15:45:46.193 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 >> v1:172.21.15.64:6791/0 conn(0x7f593c0a3000 legacy=0x7f593c1d6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:46.193 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 >> v1:172.21.15.100:6790/0 conn(0x7f593c0a2510 legacy=0x7f593c1d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:46.193 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient: _finish_hunting 0 2024-09-17T15:45:46.193 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 monclient: found mon.b 2024-09-17T15:45:46.193 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:46.193 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:45:46.193 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593c0a61d0 con 0x7f593c1439e0 2024-09-17T15:45:46.193 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient: _finish_auth 0 2024-09-17T15:45:46.193 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient: _check_auth_tickets 2024-09-17T15:45:46.193 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:46.193 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f5943059640 5 monclient: authenticate success, global_id 4151 2024-09-17T15:45:46.193 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f5943059640 10 monclient: _renew_subs 2024-09-17T15:45:46.193 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f5943059640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:45:46.194 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f5943059640 1 -- 172.21.15.64:0/427584231 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f593c12b590 con 0x7f593c1439e0 2024-09-17T15:45:46.194 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f5943059640 10 monclient: _renew_subs 2024-09-17T15:45:46.194 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f5943059640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:45:46.194 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f5943059640 1 -- 172.21.15.64:0/427584231 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f593c143240 con 0x7f593c1439e0 2024-09-17T15:45:46.194 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59340020b0 con 0x7f593c1439e0 2024-09-17T15:45:46.194 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:46.194 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5934003f60 con 0x7f593c1439e0 2024-09-17T15:45:46.194 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:46.194 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f593affd640 4 set_mon_vals no callback set 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 10 monclient: dump: 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:46.195 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:46.196 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:46.196 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:46.196 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:46.196 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:46.196 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f59340044d0 con 0x7f593c1439e0 2024-09-17T15:45:46.196 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.154+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 692025209 0 0) 0x7f5934006130 con 0x7f593c1439e0 2024-09-17T15:45:46.196 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.155+0000 7f5943059640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:46.196 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.155+0000 7f5943059640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:46.196 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.155+0000 7f5943059640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:45:46.196 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.155+0000 7f5943059640 1 -- 172.21.15.64:0/427584231 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f591c005080 con 0x7f593c1439e0 2024-09-17T15:45:46.196 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.157+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f59340047c0 con 0x7f593c1439e0 2024-09-17T15:45:46.196 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.157+0000 7f59397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:46.196 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.157+0000 7f59397fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:46.220 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.218+0000 7f5943059640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:45:46.220 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.218+0000 7f5943059640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:45:46.220 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.218+0000 7f5943059640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:45:46.220 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.218+0000 7f5943059640 1 -- 172.21.15.64:0/427584231 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f591c005370 con 0x7f593c1439e0 2024-09-17T15:45:46.220 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.219+0000 7f59397fa640 1 -- 172.21.15.64:0/427584231 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 2110007902) 0x7f5934005cd0 con 0x7f593c1439e0 2024-09-17T15:45:46.221 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.219+0000 7f59397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:45:46.221 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.219+0000 7f59397fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:46.221 INFO:teuthology.misc.health.smithi064.stdout: 2024-09-17T15:45:46.221 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":8,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","created":"2024-09-17T15:45:43.591977+0000","modified":"2024-09-17T15:45:45.218655+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T15:45:45.218655+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d9b48f9-b24c-453f-a5e9-79d74f56118c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6b8dcb8b-604c-46d3-a682-5ef3e93155d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34a266c9-43fe-4e7b-ae85-df0115c5f618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c8740bab-d046-474f-b6a4-84333eb1e094","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"08a1700b-63c5-4028-824f-dbd38931ddb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"81fa403f-39e9-44fd-8b1f-ef72fd7ba855","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:45:46.222 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.219+0000 7f5943059640 10 monclient: shutdown 2024-09-17T15:45:46.222 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.219+0000 7f5943059640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:46.222 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.219+0000 7f5943059640 1 -- 172.21.15.64:0/427584231 >> v1:172.21.15.64:6790/0 conn(0x7f593c1439e0 legacy=0x7f593c1da620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:46.222 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.219+0000 7f5943059640 1 -- 172.21.15.64:0/427584231 shutdown_connections 2024-09-17T15:45:46.222 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.219+0000 7f5943059640 1 -- 172.21.15.64:0/427584231 >> 172.21.15.64:0/427584231 conn(0x7f593c0a7f60 msgr2=0x7f593c06df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:46.222 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.220+0000 7f5943059640 1 -- 172.21.15.64:0/427584231 shutdown_connections 2024-09-17T15:45:46.222 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:46.220+0000 7f5943059640 1 -- 172.21.15.64:0/427584231 wait complete. 2024-09-17T15:45:46.230 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-17T15:45:46.440 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-17T15:45:46.438+0000 7f907861a640 -1 Falling back to public interface 2024-09-17T15:45:46.444 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-09-17T15:45:46.442+0000 7f6b565b2640 -1 Falling back to public interface 2024-09-17T15:45:46.467 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-17T15:45:46.465+0000 7f00a902c640 -1 Falling back to public interface 2024-09-17T15:45:46.476 INFO:tasks.ceph.osd.3.smithi100.stderr:2024-09-17T15:45:46.474+0000 7fccd5264640 -1 Falling back to public interface 2024-09-17T15:45:46.931 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-17T15:45:46.929+0000 7f5fa473e640 -1 Falling back to public interface 2024-09-17T15:45:46.976 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-17T15:45:46.974+0000 7fe373085640 -1 Falling back to public interface 2024-09-17T15:45:48.845 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-17T15:45:48.841+0000 7f9e96146640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-17T15:45:48.845 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-17T15:45:48.842+0000 7f9e96146640 -1 mgr.server handle_report got status from non-daemon mon.f 2024-09-17T15:45:48.845 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-17T15:45:48.842+0000 7f9e96146640 -1 mgr.server handle_report got status from non-daemon mon.h 2024-09-17T15:45:48.846 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-17T15:45:48.842+0000 7f9e96146640 -1 mgr.server handle_report got status from non-daemon mon.g 2024-09-17T15:45:48.846 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-17T15:45:48.842+0000 7f9e96146640 -1 mgr.server handle_report got status from non-daemon mon.i 2024-09-17T15:45:48.846 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-17T15:45:48.842+0000 7f9e96146640 -1 mgr.server handle_report got status from non-daemon mon.e 2024-09-17T15:45:48.846 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-17T15:45:48.843+0000 7f9e96146640 -1 mgr.server handle_report got status from non-daemon mon.d 2024-09-17T15:45:48.846 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-17T15:45:48.845+0000 7f9e96146640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-17T15:45:50.250 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-17T15:45:50.248+0000 7f907861a640 -1 osd.1 0 log_to_monitors true 2024-09-17T15:45:50.268 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-17T15:45:50.266+0000 7f00a902c640 -1 osd.5 0 log_to_monitors true 2024-09-17T15:45:50.304 INFO:tasks.ceph.osd.3.smithi100.stderr:2024-09-17T15:45:50.302+0000 7fccd5264640 -1 osd.3 0 log_to_monitors true 2024-09-17T15:45:50.474 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-09-17T15:45:50.472+0000 7f6b565b2640 -1 osd.2 0 log_to_monitors true 2024-09-17T15:45:50.496 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-17T15:45:50.494+0000 7fe373085640 -1 osd.4 0 log_to_monitors true 2024-09-17T15:45:50.673 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-17T15:45:50.671+0000 7f5fa473e640 -1 osd.0 0 log_to_monitors true 2024-09-17T15:45:51.872 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-17T15:45:51.869+0000 7f00a4fbb640 -1 osd.5 0 waiting for initial osdmap 2024-09-17T15:45:51.872 INFO:tasks.ceph.osd.3.smithi100.stderr:2024-09-17T15:45:51.870+0000 7fccd11f3640 -1 osd.3 0 waiting for initial osdmap 2024-09-17T15:45:51.876 INFO:tasks.ceph.osd.3.smithi100.stderr:2024-09-17T15:45:51.874+0000 7fcccd01a640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T15:45:51.876 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-17T15:45:51.874+0000 7f00a0de2640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T15:45:52.836 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T15:45:52.940 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.939+0000 7fc468fec640 10 monclient: get_monmap_and_config 2024-09-17T15:45:52.941 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.939+0000 7fc468fec640 10 monclient: build_initial_monmap 2024-09-17T15:45:52.941 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.939+0000 7fc468fec640 10 monclient: monmap: 2024-09-17T15:45:52.941 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-09-17T15:45:52.941 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:52.941 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:52.940056+0000 2024-09-17T15:45:52.941 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:52.940056+0000 2024-09-17T15:45:52.941 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:52.941 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:52.941 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:52.941 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:52.941 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:52.942 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:52.942 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:52.942 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:52.942 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:52.942 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:52.942 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:52.942 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:52.944 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.939+0000 7fc468fec640 1 Processor -- start 2024-09-17T15:45:52.945 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.939+0000 7fc468fec640 1 -- start start 2024-09-17T15:45:52.945 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.939+0000 7fc468fec640 10 monclient: init 2024-09-17T15:45:52.945 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.940+0000 7fc468fec640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:52.945 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.940+0000 7fc468fec640 10 monclient: _add_conns ranks=[4,1,6,3,8,0,7,5,2] 2024-09-17T15:45:52.945 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.940+0000 7fc468fec640 10 monclient(hunting): picked mon.noname-c con 0x7fc4640a5a10 addr v1:172.21.15.64:6791/0 2024-09-17T15:45:52.945 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.940+0000 7fc468fec640 10 monclient(hunting): picked mon.noname-f con 0x7fc4640a6500 addr v1:172.21.15.100:6789/0 2024-09-17T15:45:52.945 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.940+0000 7fc468fec640 10 monclient(hunting): picked mon.noname-d con 0x7fc46413f5d0 addr v1:172.21.15.64:6792/0 2024-09-17T15:45:52.945 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.940+0000 7fc468fec640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4640a3ee0 con 0x7fc4640a6500 2024-09-17T15:45:52.945 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.940+0000 7fc468fec640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4640a70a0 con 0x7fc4640a5a10 2024-09-17T15:45:52.945 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.940+0000 7fc468fec640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4640a2c30 con 0x7fc46413f5d0 2024-09-17T15:45:52.945 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.940+0000 7fc468fec640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:52.945 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.940+0000 7fc468fec640 10 monclient(hunting): authenticate will time out at 750.615680s 2024-09-17T15:45:52.946 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.940+0000 7fc462d9d640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fc46413f5d0 0x7fc4641419c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:60992/0 (socket says 172.21.15.64:60992) 2024-09-17T15:45:52.946 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.940+0000 7fc462d9d640 1 -- 172.21.15.64:0/4097446112 learned_addr learned my addr 172.21.15.64:0/4097446112 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:45:52.946 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 -- 172.21.15.64:0/4097446112 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583129889 0 0) 0x7fc4640a2c30 con 0x7fc46413f5d0 2024-09-17T15:45:52.946 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient(hunting): _negotiate 2024-09-17T15:45:52.946 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:52.946 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:52.946 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient(hunting): my global_id is 4209 2024-09-17T15:45:52.946 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 -- 172.21.15.64:0/4097446112 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc44c003520 con 0x7fc46413f5d0 2024-09-17T15:45:52.946 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 -- 172.21.15.64:0/4097446112 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 656259990 0 0) 0x7fc4640a70a0 con 0x7fc4640a5a10 2024-09-17T15:45:52.946 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient(hunting): _negotiate 2024-09-17T15:45:52.946 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:52.947 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:52.947 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient(hunting): my global_id is 4234 2024-09-17T15:45:52.947 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 -- 172.21.15.64:0/4097446112 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4640a2c30 con 0x7fc4640a5a10 2024-09-17T15:45:52.947 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 -- 172.21.15.64:0/4097446112 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 359621902 0 0) 0x7fc4640a3ee0 con 0x7fc4640a6500 2024-09-17T15:45:52.947 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient(hunting): _negotiate 2024-09-17T15:45:52.947 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:52.947 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:52.947 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient(hunting): my global_id is 4258 2024-09-17T15:45:52.947 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 -- 172.21.15.64:0/4097446112 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4640a70a0 con 0x7fc4640a6500 2024-09-17T15:45:52.947 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 -- 172.21.15.64:0/4097446112 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 842780215 0 0) 0x7fc44c003520 con 0x7fc46413f5d0 2024-09-17T15:45:52.947 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 -- 172.21.15.64:0/4097446112 >> v1:172.21.15.64:6791/0 conn(0x7fc4640a5a10 legacy=0x7fc4640a5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:52.947 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 -- 172.21.15.64:0/4097446112 >> v1:172.21.15.100:6789/0 conn(0x7fc4640a6500 legacy=0x7fc46413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:52.947 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient: _finish_hunting 0 2024-09-17T15:45:52.948 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 monclient: found mon.noname-d 2024-09-17T15:45:52.948 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:52.948 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:45:52.948 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 -- 172.21.15.64:0/4097446112 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4640a2f10 con 0x7fc46413f5d0 2024-09-17T15:45:52.948 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient: _finish_auth 0 2024-09-17T15:45:52.948 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient: _check_auth_tickets 2024-09-17T15:45:52.948 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:52.948 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 -- 172.21.15.64:0/4097446112 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc45c001bb0 con 0x7fc46413f5d0 2024-09-17T15:45:52.948 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc468fec640 5 monclient: authenticate success, global_id 4209 2024-09-17T15:45:52.948 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:52.948 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:45:52.948 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient: dump: 2024-09-17T15:45:52.948 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:52.949 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:52.949 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:52.949 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:52.949 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:52.949 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:52.949 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:52.949 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:52.949 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:52.949 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:52.949 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:52.949 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:52.949 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:52.949 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc468fec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 -- 172.21.15.64:0/4097446112 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc45c001dd0 con 0x7fc46413f5d0 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc468fec640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc468fec640 4 set_mon_vals no callback set 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 1 -- 172.21.15.64:0/4097446112 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc45c004f60 con 0x7fc46413f5d0 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.941+0000 7fc46159a640 10 monclient: dump: 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:52.950 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:52.951 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.942+0000 7fc468fec640 10 monclient: shutdown 2024-09-17T15:45:52.952 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.942+0000 7fc468fec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:52.952 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.942+0000 7fc468fec640 1 -- 172.21.15.64:0/4097446112 >> v1:172.21.15.64:6792/0 conn(0x7fc46413f5d0 legacy=0x7fc4641419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:52.952 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.942+0000 7fc468fec640 1 -- 172.21.15.64:0/4097446112 shutdown_connections 2024-09-17T15:45:52.952 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.942+0000 7fc468fec640 1 -- 172.21.15.64:0/4097446112 >> 172.21.15.64:0/4097446112 conn(0x7fc4640a7b10 msgr2=0x7fc4640a7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:52.952 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.942+0000 7fc468fec640 1 -- 172.21.15.64:0/4097446112 shutdown_connections 2024-09-17T15:45:52.952 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.942+0000 7fc468fec640 1 -- 172.21.15.64:0/4097446112 wait complete. 2024-09-17T15:45:52.952 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.943+0000 7fc468fec640 10 monclient: build_initial_monmap 2024-09-17T15:45:52.952 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.943+0000 7fc468fec640 10 monclient: monmap: 2024-09-17T15:45:52.952 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-09-17T15:45:52.952 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:52.952 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-09-17T15:45:52.952 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-09-17T15:45:52.952 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.943+0000 7fc468fec640 1 Processor -- start 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.943+0000 7fc468fec640 1 -- start start 2024-09-17T15:45:52.953 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.943+0000 7fc468fec640 10 monclient: init 2024-09-17T15:45:52.957 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc468fec640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:52.957 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc468fec640 10 monclient: _add_conns ranks=[8,0,7,1,5,6,2,4,3] 2024-09-17T15:45:52.957 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc468fec640 10 monclient(hunting): picked mon.noname-e con 0x7fc4640a5a10 addr v1:172.21.15.64:6793/0 2024-09-17T15:45:52.957 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc468fec640 10 monclient(hunting): picked mon.noname-a con 0x7fc4640a6500 addr v1:172.21.15.64:6789/0 2024-09-17T15:45:52.958 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc468fec640 10 monclient(hunting): picked mon.noname-i con 0x7fc46413f5d0 addr v1:172.21.15.100:6792/0 2024-09-17T15:45:52.958 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc468fec640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc464138d70 con 0x7fc4640a6500 2024-09-17T15:45:52.958 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc468fec640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc464123d20 con 0x7fc46413f5d0 2024-09-17T15:45:52.958 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc46259c640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fc4640a5a10 0x7fc464121660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:40118/0 (socket says 172.21.15.64:40118) 2024-09-17T15:45:52.958 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc46259c640 1 -- 172.21.15.64:0/1168156272 learned_addr learned my addr 172.21.15.64:0/1168156272 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:45:52.958 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc461d9b640 1 --1- 172.21.15.64:0/1168156272 >> v1:172.21.15.64:6789/0 conn(0x7fc4640a6500 0x7fc464121d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:38642/0 (socket says 172.21.15.64:38642) 2024-09-17T15:45:52.958 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc468fec640 1 -- 172.21.15.64:0/1168156272 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc464122480 con 0x7fc4640a5a10 2024-09-17T15:45:52.958 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc468fec640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:52.958 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc468fec640 10 monclient(hunting): authenticate will time out at 750.619331s 2024-09-17T15:45:52.959 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2096154787 0 0) 0x7fc464138d70 con 0x7fc4640a6500 2024-09-17T15:45:52.959 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:45:52.959 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:52.959 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:52.959 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): my global_id is 4203 2024-09-17T15:45:52.959 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc444003520 con 0x7fc4640a6500 2024-09-17T15:45:52.959 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4276270781 0 0) 0x7fc464122480 con 0x7fc4640a5a10 2024-09-17T15:45:52.959 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:45:52.959 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:52.960 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:52.960 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): my global_id is 4184 2024-09-17T15:45:52.960 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc464138d70 con 0x7fc4640a5a10 2024-09-17T15:45:52.960 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 496333082 0 0) 0x7fc464123d20 con 0x7fc46413f5d0 2024-09-17T15:45:52.960 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:45:52.960 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:52.960 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:52.960 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): my global_id is 4228 2024-09-17T15:45:52.961 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc464122480 con 0x7fc46413f5d0 2024-09-17T15:45:52.961 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 681684737 0 0) 0x7fc444003520 con 0x7fc4640a6500 2024-09-17T15:45:52.961 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc464123d20 con 0x7fc4640a6500 2024-09-17T15:45:52.961 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc458001b10 con 0x7fc4640a6500 2024-09-17T15:45:52.961 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:52.961 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:45:52.961 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): dump: 2024-09-17T15:45:52.961 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:52.961 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:52.962 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:52.962 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:52.962 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:52.962 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:52.962 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:52.962 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:52.962 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:52.962 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:52.962 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:52.962 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:52.962 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:52.962 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:52.963 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:52.963 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:52.963 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:45:52.963 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3718147094 0 0) 0x7fc464122480 con 0x7fc46413f5d0 2024-09-17T15:45:52.963 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc444003520 con 0x7fc46413f5d0 2024-09-17T15:45:52.963 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1473779520 0 0) 0x7fc464138d70 con 0x7fc4640a5a10 2024-09-17T15:45:52.963 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc464122480 con 0x7fc4640a5a10 2024-09-17T15:45:52.963 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc45c0013a0 con 0x7fc46413f5d0 2024-09-17T15:45:52.963 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.944+0000 7fc452ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:52.963 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:45:52.963 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 10 monclient(hunting): dump: 2024-09-17T15:45:52.963 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:52.964 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:52.965 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:52.965 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc454001a40 con 0x7fc4640a5a10 2024-09-17T15:45:52.965 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:52.965 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:45:52.965 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 10 monclient(hunting): dump: 2024-09-17T15:45:52.965 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:52.965 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:52.965 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:52.965 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:52.965 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:52.965 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:52.965 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:52.966 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:52.966 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:52.966 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:52.966 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:52.966 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:52.966 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:52.966 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:52.966 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:52.966 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:52.966 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2099855127 0 0) 0x7fc464123d20 con 0x7fc4640a6500 2024-09-17T15:45:52.966 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 >> v1:172.21.15.64:6793/0 conn(0x7fc4640a5a10 legacy=0x7fc464121660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:52.967 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 >> v1:172.21.15.100:6792/0 conn(0x7fc46413f5d0 legacy=0x7fc4641d5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:52.967 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:45:52.967 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 1 monclient: found mon.a 2024-09-17T15:45:52.967 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:52.967 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:45:52.967 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc464124000 con 0x7fc4640a6500 2024-09-17T15:45:52.967 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 10 monclient: _finish_auth 0 2024-09-17T15:45:52.967 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:45:52.967 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:52.968 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc468fec640 5 monclient: authenticate success, global_id 4203 2024-09-17T15:45:52.968 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc468fec640 10 monclient: _renew_subs 2024-09-17T15:45:52.968 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc468fec640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:45:52.968 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc468fec640 1 -- 172.21.15.64:0/1168156272 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc46406e970 con 0x7fc4640a6500 2024-09-17T15:45:52.968 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc468fec640 10 monclient: _renew_subs 2024-09-17T15:45:52.968 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc468fec640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:45:52.968 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc468fec640 1 -- 172.21.15.64:0/1168156272 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc464123a40 con 0x7fc4640a6500 2024-09-17T15:45:52.968 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc4580038c0 con 0x7fc4640a6500 2024-09-17T15:45:52.968 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:52.968 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc458003df0 con 0x7fc4640a6500 2024-09-17T15:45:52.968 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:52.968 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc460d99640 4 set_mon_vals no callback set 2024-09-17T15:45:52.968 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:52.969 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.945+0000 7fc452ffd640 10 monclient: dump: 2024-09-17T15:45:52.969 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:52.969 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:52.969 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:52.969 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:52.969 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:52.969 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:52.969 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:52.969 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.946+0000 7fc468fec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.946+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) ==== 50940+0+0 (unknown 9449506 0 0) 0x7fc458004050 con 0x7fc4640a6500 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.946+0000 7fc468fec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.946+0000 7fc468fec640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.946+0000 7fc468fec640 1 -- 172.21.15.64:0/1168156272 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc448005080 con 0x7fc4640a6500 2024-09-17T15:45:52.970 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.946+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(11..11 src has 1..11) ==== 2207+0+0 (unknown 148532211 0 0) 0x7fc45804e490 con 0x7fc4640a6500 2024-09-17T15:45:52.971 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.947+0000 7fc46259c640 10 monclient: get_auth_request con 0x7fc444040df0 auth_method 0 2024-09-17T15:45:52.971 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.954+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fc4580176e0 con 0x7fc4640a6500 2024-09-17T15:45:52.971 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.954+0000 7fc452ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:52.971 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:52.954+0000 7fc452ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:53.158 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.156+0000 7fc468fec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:45:53.158 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.156+0000 7fc468fec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:45:53.158 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.156+0000 7fc468fec640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:45:53.158 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.156+0000 7fc468fec640 1 -- 172.21.15.64:0/1168156272 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc448005370 con 0x7fc4640a6500 2024-09-17T15:45:53.159 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.157+0000 7fc452ffd640 1 -- 172.21.15.64:0/1168156272 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+6113 (unknown 238102980 0 3118323046) 0x7fc458016fe0 con 0x7fc4640a6500 2024-09-17T15:45:53.159 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.157+0000 7fc452ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:45:53.159 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.157+0000 7fc452ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:53.159 INFO:teuthology.misc.health.smithi064.stdout: 2024-09-17T15:45:53.160 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","created":"2024-09-17T15:45:43.591977+0000","modified":"2024-09-17T15:45:52.863756+0000","last_up_change":"2024-09-17T15:45:52.863756+0000","last_in_change":"2024-09-17T15:45:45.218655+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1d9b48f9-b24c-453f-a5e9-79d74f56118c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6b8dcb8b-604c-46d3-a682-5ef3e93155d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34a266c9-43fe-4e7b-ae85-df0115c5f618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c8740bab-d046-474f-b6a4-84333eb1e094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6804","nonce":2791828047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6805","nonce":2791828047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6807","nonce":2791828047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6806","nonce":2791828047}]},"public_addr":"172.21.15.100:6804/2791828047","cluster_addr":"172.21.15.100:6805/2791828047","heartbeat_back_addr":"172.21.15.100:6807/2791828047","heartbeat_front_addr":"172.21.15.100:6806/2791828047","state":["exists","up"]},{"osd":4,"uuid":"08a1700b-63c5-4028-824f-dbd38931ddb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"81fa403f-39e9-44fd-8b1f-ef72fd7ba855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6800","nonce":4145568194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6801","nonce":4145568194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6803","nonce":4145568194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6802","nonce":4145568194}]},"public_addr":"172.21.15.100:6800/4145568194","cluster_addr":"172.21.15.100:6801/4145568194","heartbeat_back_addr":"172.21.15.100:6803/4145568194","heartbeat_front_addr":"172.21.15.100:6802/4145568194","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:45:53.161 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.159+0000 7fc468fec640 1 -- 172.21.15.64:0/1168156272 >> v1:172.21.15.100:6812/4210963484 conn(0x7fc444040df0 legacy=0x7fc4440432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:53.161 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.159+0000 7fc468fec640 10 monclient: shutdown 2024-09-17T15:45:53.161 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.159+0000 7fc468fec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:53.161 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.159+0000 7fc468fec640 1 -- 172.21.15.64:0/1168156272 >> v1:172.21.15.64:6789/0 conn(0x7fc4640a6500 legacy=0x7fc464121d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:53.162 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.159+0000 7fc468fec640 1 -- 172.21.15.64:0/1168156272 shutdown_connections 2024-09-17T15:45:53.162 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.159+0000 7fc468fec640 1 -- 172.21.15.64:0/1168156272 >> 172.21.15.64:0/1168156272 conn(0x7fc4640a7b10 msgr2=0x7fc4640a8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:53.162 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.159+0000 7fc468fec640 1 -- 172.21.15.64:0/1168156272 shutdown_connections 2024-09-17T15:45:53.162 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:53.159+0000 7fc468fec640 1 -- 172.21.15.64:0/1168156272 wait complete. 2024-09-17T15:45:53.171 DEBUG:teuthology.misc:2 of 6 OSDs are up 2024-09-17T15:45:54.495 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-17T15:45:54.493+0000 7fe36f014640 -1 osd.4 0 waiting for initial osdmap 2024-09-17T15:45:54.498 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-17T15:45:54.496+0000 7fe366e33640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T15:45:54.965 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-09-17T15:45:54.962+0000 7f6b52541640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T15:45:54.973 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-09-17T15:45:54.971+0000 7f6b4a360640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T15:45:55.020 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-17T15:45:55.018+0000 7f5fa06cd640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T15:45:55.025 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-17T15:45:55.023+0000 7f5f984ec640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T15:45:55.040 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-17T15:45:55.038+0000 7f90745a9640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T15:45:55.045 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-09-17T15:45:55.044+0000 7f906c3c8640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T15:45:59.778 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T15:45:59.881 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.880+0000 7f17b7ff4640 10 monclient: get_monmap_and_config 2024-09-17T15:45:59.881 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.880+0000 7f17b7ff4640 10 monclient: build_initial_monmap 2024-09-17T15:45:59.881 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.880+0000 7f17b7ff4640 10 monclient: monmap: 2024-09-17T15:45:59.882 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-09-17T15:45:59.882 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:59.882 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:59.880895+0000 2024-09-17T15:45:59.882 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:59.880895+0000 2024-09-17T15:45:59.882 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:59.882 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:59.882 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:59.882 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:59.882 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:59.882 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:59.882 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:59.882 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:59.882 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:59.883 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:59.883 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:59.883 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:59.891 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.880+0000 7f17b7ff4640 1 Processor -- start 2024-09-17T15:45:59.891 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.880+0000 7f17b7ff4640 1 -- start start 2024-09-17T15:45:59.891 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.880+0000 7f17b7ff4640 10 monclient: init 2024-09-17T15:45:59.891 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b7ff4640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:59.891 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b7ff4640 10 monclient: _add_conns ranks=[1,2,4,7,0,3,5,8,6] 2024-09-17T15:45:59.891 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b7ff4640 10 monclient(hunting): picked mon.noname-f con 0x7f17b00a5b10 addr v1:172.21.15.100:6789/0 2024-09-17T15:45:59.891 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b7ff4640 10 monclient(hunting): picked mon.noname-b con 0x7f17b00a6600 addr v1:172.21.15.64:6790/0 2024-09-17T15:45:59.891 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b7ff4640 10 monclient(hunting): picked mon.noname-c con 0x7f17b009dc60 addr v1:172.21.15.64:6791/0 2024-09-17T15:45:59.891 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b7ff4640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b00a6c40 con 0x7f17b00a5b10 2024-09-17T15:45:59.891 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b7ff4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b009cfa0 con 0x7f17b00a6600 2024-09-17T15:45:59.891 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b7ff4640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b009e0d0 con 0x7f17b009dc60 2024-09-17T15:45:59.891 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b7ff4640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:59.892 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b7ff4640 10 monclient(hunting): authenticate will time out at 757.556469s 2024-09-17T15:45:59.892 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b558f640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f17b00a6600 0x7f17b013dee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:48156/0 (socket says 172.21.15.64:48156) 2024-09-17T15:45:59.892 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b558f640 1 -- 172.21.15.64:0/3565659920 learned_addr learned my addr 172.21.15.64:0/3565659920 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:45:59.892 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b6591640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f17b009dc60 0x7f17b0141610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:50594/0 (socket says 172.21.15.64:50594) 2024-09-17T15:45:59.892 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 1 -- 172.21.15.64:0/3565659920 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1156147216 0 0) 0x7f17b009cfa0 con 0x7f17b00a6600 2024-09-17T15:45:59.892 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 10 monclient(hunting): _negotiate 2024-09-17T15:45:59.892 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:59.892 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:59.892 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 10 monclient(hunting): my global_id is 4187 2024-09-17T15:45:59.892 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 1 -- 172.21.15.64:0/3565659920 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f179c003520 con 0x7f17b00a6600 2024-09-17T15:45:59.892 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 1 -- 172.21.15.64:0/3565659920 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2336260735 0 0) 0x7f17b009e0d0 con 0x7f17b009dc60 2024-09-17T15:45:59.892 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 10 monclient(hunting): _negotiate 2024-09-17T15:45:59.893 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:59.893 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:59.893 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 10 monclient(hunting): my global_id is 4261 2024-09-17T15:45:59.893 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 1 -- 172.21.15.64:0/3565659920 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17b009cfa0 con 0x7f17b009dc60 2024-09-17T15:45:59.893 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 1 -- 172.21.15.64:0/3565659920 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1814971971 0 0) 0x7f17b00a6c40 con 0x7f17b00a5b10 2024-09-17T15:45:59.893 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 10 monclient(hunting): _negotiate 2024-09-17T15:45:59.893 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:59.893 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:59.893 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 10 monclient(hunting): my global_id is 4285 2024-09-17T15:45:59.893 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.881+0000 7f17b4d8e640 1 -- 172.21.15.64:0/3565659920 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17b009e0d0 con 0x7f17b00a5b10 2024-09-17T15:45:59.893 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 1 -- 172.21.15.64:0/3565659920 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 770099466 0 0) 0x7f179c003520 con 0x7f17b00a6600 2024-09-17T15:45:59.893 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 1 -- 172.21.15.64:0/3565659920 >> v1:172.21.15.64:6791/0 conn(0x7f17b009dc60 legacy=0x7f17b0141610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:59.893 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 1 -- 172.21.15.64:0/3565659920 >> v1:172.21.15.100:6789/0 conn(0x7f17b00a5b10 legacy=0x7f17b00a5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:59.894 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 10 monclient: _finish_hunting 0 2024-09-17T15:45:59.894 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 1 monclient: found mon.noname-b 2024-09-17T15:45:59.894 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:59.894 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:45:59.894 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 1 -- 172.21.15.64:0/3565659920 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b00a7360 con 0x7f17b00a6600 2024-09-17T15:45:59.894 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 10 monclient: _finish_auth 0 2024-09-17T15:45:59.894 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 10 monclient: _check_auth_tickets 2024-09-17T15:45:59.894 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:59.894 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 1 -- 172.21.15.64:0/3565659920 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f17a0001830 con 0x7f17b00a6600 2024-09-17T15:45:59.894 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:59.894 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:45:59.894 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 10 monclient: dump: 2024-09-17T15:45:59.894 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:59.895 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:59.895 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:59.895 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:59.895 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:59.895 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:59.895 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:59.895 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:59.895 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:59.895 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:59.895 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:59.895 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:59.895 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:59.895 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:59.896 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:59.896 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:59.896 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b7ff4640 5 monclient: authenticate success, global_id 4187 2024-09-17T15:45:59.897 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b7ff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:59.897 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 1 -- 172.21.15.64:0/3565659920 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17a00019d0 con 0x7f17b00a6600 2024-09-17T15:45:59.897 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:59.897 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 1 -- 172.21.15.64:0/3565659920 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f17a0004be0 con 0x7f17b00a6600 2024-09-17T15:45:59.897 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b7ff4640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:59.897 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b7ff4640 4 set_mon_vals no callback set 2024-09-17T15:45:59.897 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:59.897 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:59.897 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b4d8e640 10 monclient: dump: 2024-09-17T15:45:59.897 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:59.897 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:59.898 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:59.898 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:59.898 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:59.898 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:59.898 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:59.898 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:59.898 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:59.898 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:59.898 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:59.898 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:59.898 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:59.898 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:59.898 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:59.899 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:59.899 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b7ff4640 10 monclient: shutdown 2024-09-17T15:45:59.899 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b7ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:59.899 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3565659920 >> v1:172.21.15.64:6790/0 conn(0x7f17b00a6600 legacy=0x7f17b013dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:59.899 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3565659920 shutdown_connections 2024-09-17T15:45:59.899 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.882+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3565659920 >> 172.21.15.64:0/3565659920 conn(0x7f17b00a8030 msgr2=0x7f17b00a8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:59.899 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.883+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3565659920 shutdown_connections 2024-09-17T15:45:59.899 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.883+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3565659920 wait complete. 2024-09-17T15:45:59.899 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.883+0000 7f17b7ff4640 10 monclient: build_initial_monmap 2024-09-17T15:45:59.899 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.883+0000 7f17b7ff4640 10 monclient: monmap: 2024-09-17T15:45:59.899 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-09-17T15:45:59.899 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:59.899 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-09-17T15:45:59.900 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-09-17T15:45:59.900 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:59.900 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-09-17T15:45:59.900 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:45:59.900 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:45:59.900 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:45:59.900 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:45:59.900 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:45:59.900 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:45:59.900 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:45:59.900 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:45:59.900 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:45:59.900 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:59.901 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.883+0000 7f17b7ff4640 1 Processor -- start 2024-09-17T15:45:59.901 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b7ff4640 1 -- start start 2024-09-17T15:45:59.901 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b7ff4640 10 monclient: init 2024-09-17T15:45:59.901 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b7ff4640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:59.901 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b7ff4640 10 monclient: _add_conns ranks=[8,4,1,6,2,3,0,7,5] 2024-09-17T15:45:59.901 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b7ff4640 10 monclient(hunting): picked mon.noname-e con 0x7f17b009dc60 addr v1:172.21.15.64:6793/0 2024-09-17T15:45:59.901 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b7ff4640 10 monclient(hunting): picked mon.noname-c con 0x7f17b00a5b10 addr v1:172.21.15.64:6791/0 2024-09-17T15:45:59.901 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b7ff4640 10 monclient(hunting): picked mon.noname-f con 0x7f17b00a6600 addr v1:172.21.15.100:6789/0 2024-09-17T15:45:59.901 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b7ff4640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b0128a30 con 0x7f17b00a6600 2024-09-17T15:45:59.901 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b7ff4640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b0128470 con 0x7f17b00a5b10 2024-09-17T15:45:59.902 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b7ff4640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b0128750 con 0x7f17b009dc60 2024-09-17T15:45:59.902 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b7ff4640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:59.903 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b7ff4640 10 monclient(hunting): authenticate will time out at 757.559584s 2024-09-17T15:45:59.903 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b558f640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f17b00a5b10 0x7f17b01d2570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:50596/0 (socket says 172.21.15.64:50596) 2024-09-17T15:45:59.903 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b5d90640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f17b009dc60 0x7f17b0141390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:39814/0 (socket says 172.21.15.64:39814) 2024-09-17T15:45:59.903 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17b558f640 1 -- 172.21.15.64:0/3055225924 learned_addr learned my addr 172.21.15.64:0/3055225924 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:45:59.903 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2249884913 0 0) 0x7f17b0128470 con 0x7f17b00a5b10 2024-09-17T15:45:59.903 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17a67fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:59.903 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:59.903 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:59.903 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17a67fc640 10 monclient(hunting): my global_id is 4270 2024-09-17T15:45:59.903 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1794003520 con 0x7f17b00a5b10 2024-09-17T15:45:59.903 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1775094114 0 0) 0x7f17b0128750 con 0x7f17b009dc60 2024-09-17T15:45:59.903 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17a67fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:59.903 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:59.904 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:59.904 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17a67fc640 10 monclient(hunting): my global_id is 4202 2024-09-17T15:45:59.904 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.884+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17b0128470 con 0x7f17b009dc60 2024-09-17T15:45:59.904 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317146677 0 0) 0x7f17b0128a30 con 0x7f17b00a6600 2024-09-17T15:45:59.904 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): _negotiate 2024-09-17T15:45:59.904 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:59.904 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:59.904 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): my global_id is 4294 2024-09-17T15:45:59.904 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17b0128750 con 0x7f17b00a6600 2024-09-17T15:45:59.904 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1550445389 0 0) 0x7f1794003520 con 0x7f17b00a5b10 2024-09-17T15:45:59.904 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17b0128a30 con 0x7f17b00a5b10 2024-09-17T15:45:59.904 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f17a0001830 con 0x7f17b00a5b10 2024-09-17T15:45:59.904 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): dump: 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:59.905 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:59.906 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:59.906 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:59.906 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:59.906 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:59.906 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:45:59.906 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 731302764 0 0) 0x7f17b0128470 con 0x7f17b009dc60 2024-09-17T15:45:59.906 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1794003520 con 0x7f17b009dc60 2024-09-17T15:45:59.906 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f17ac001550 con 0x7f17b009dc60 2024-09-17T15:45:59.906 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:59.906 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:45:59.906 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): dump: 2024-09-17T15:45:59.906 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:59.906 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:59.907 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:59.908 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1917403652 0 0) 0x7f17b0128750 con 0x7f17b00a6600 2024-09-17T15:45:59.908 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17b0128470 con 0x7f17b00a6600 2024-09-17T15:45:59.908 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f17a80018d0 con 0x7f17b00a6600 2024-09-17T15:45:59.908 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:45:59.908 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:45:59.908 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient(hunting): dump: 2024-09-17T15:45:59.908 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:59.908 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:59.908 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:59.908 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:59.908 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:59.908 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:59.909 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:59.909 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:59.909 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:59.909 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:59.909 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:59.909 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:59.909 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:59.909 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:59.909 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:59.910 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:59.910 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 300272067 0 0) 0x7f17b0128a30 con 0x7f17b00a5b10 2024-09-17T15:45:59.910 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 >> v1:172.21.15.64:6793/0 conn(0x7f17b009dc60 legacy=0x7f17b0141390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:59.910 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 >> v1:172.21.15.100:6789/0 conn(0x7f17b00a6600 legacy=0x7f17b01d5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:59.910 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient: _finish_hunting 0 2024-09-17T15:45:59.910 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 monclient: found mon.c 2024-09-17T15:45:59.910 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:59.910 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:59.910 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b0141aa0 con 0x7f17b00a5b10 2024-09-17T15:45:59.910 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient: _finish_auth 0 2024-09-17T15:45:59.910 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient: _check_auth_tickets 2024-09-17T15:45:59.910 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:59.910 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17b7ff4640 5 monclient: authenticate success, global_id 4270 2024-09-17T15:45:59.911 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17b7ff4640 10 monclient: _renew_subs 2024-09-17T15:45:59.911 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17b7ff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:59.911 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3055225924 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f17b0127510 con 0x7f17b00a5b10 2024-09-17T15:45:59.911 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17b7ff4640 10 monclient: _renew_subs 2024-09-17T15:45:59.911 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17b7ff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:59.911 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3055225924 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f17b01255e0 con 0x7f17b00a5b10 2024-09-17T15:45:59.911 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17a0001190 con 0x7f17b00a5b10 2024-09-17T15:45:59.911 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:59.911 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f17a0004e50 con 0x7f17b00a5b10 2024-09-17T15:45:59.911 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:59.912 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a7fff640 4 set_mon_vals no callback set 2024-09-17T15:45:59.912 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:59.912 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.885+0000 7f17a67fc640 10 monclient: dump: 2024-09-17T15:45:59.912 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-09-17T15:45:59.912 INFO:teuthology.misc.health.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:45:59.912 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:59.912 INFO:teuthology.misc.health.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:45:59.912 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:59.912 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr: 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.886+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f17a0004ff0 con 0x7f17b00a5b10 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.886+0000 7f17b7ff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.886+0000 7f17b7ff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:59.913 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.886+0000 7f17b7ff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:45:59.914 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.886+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3055225924 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1798005080 con 0x7f17b00a5b10 2024-09-17T15:45:59.914 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.887+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (unknown 3963117770 0 0) 0x7f17a004e730 con 0x7f17b00a5b10 2024-09-17T15:45:59.914 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.887+0000 7f17b5d90640 10 monclient: get_auth_request con 0x7f1794040ce0 auth_method 0 2024-09-17T15:45:59.914 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.891+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f17a0017730 con 0x7f17b00a5b10 2024-09-17T15:45:59.914 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.891+0000 7f17a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:59.914 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:45:59.891+0000 7f17a67fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:00.081 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.079+0000 7f17b7ff4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:46:00.081 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.080+0000 7f17b7ff4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:46:00.081 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.080+0000 7f17b7ff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:00.082 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.080+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3055225924 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1798005370 con 0x7f17b00a5b10 2024-09-17T15:46:00.082 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.080+0000 7f17a67fc640 1 -- 172.21.15.64:0/3055225924 <== mon.4 v1:172.21.15.64:6791/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+9049 (unknown 4221056075 0 976659260) 0x7f17a0017030 con 0x7f17b00a5b10 2024-09-17T15:46:00.082 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.080+0000 7f17a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:46:00.082 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.080+0000 7f17a67fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:00.082 INFO:teuthology.misc.health.smithi064.stdout: 2024-09-17T15:46:00.083 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":15,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","created":"2024-09-17T15:45:43.591977+0000","modified":"2024-09-17T15:45:56.899065+0000","last_up_change":"2024-09-17T15:45:55.895333+0000","last_in_change":"2024-09-17T15:45:45.218655+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T15:45:53.848072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1d9b48f9-b24c-453f-a5e9-79d74f56118c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":796819178}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":796819178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":796819178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":796819178}]},"public_addr":"172.21.15.64:6808/796819178","cluster_addr":"172.21.15.64:6809/796819178","heartbeat_back_addr":"172.21.15.64:6811/796819178","heartbeat_front_addr":"172.21.15.64:6810/796819178","state":["exists","up"]},{"osd":1,"uuid":"6b8dcb8b-604c-46d3-a682-5ef3e93155d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6800","nonce":3889666033}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":3889666033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":3889666033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":3889666033}]},"public_addr":"172.21.15.64:6800/3889666033","cluster_addr":"172.21.15.64:6801/3889666033","heartbeat_back_addr":"172.21.15.64:6803/3889666033","heartbeat_front_addr":"172.21.15.64:6802/3889666033","state":["exists","up"]},{"osd":2,"uuid":"34a266c9-43fe-4e7b-ae85-df0115c5f618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":1825798494}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":1825798494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":1825798494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":1825798494}]},"public_addr":"172.21.15.64:6804/1825798494","cluster_addr":"172.21.15.64:6805/1825798494","heartbeat_back_addr":"172.21.15.64:6807/1825798494","heartbeat_front_addr":"172.21.15.64:6806/1825798494","state":["exists","up"]},{"osd":3,"uuid":"c8740bab-d046-474f-b6a4-84333eb1e094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6804","nonce":2791828047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6805","nonce":2791828047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6807","nonce":2791828047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6806","nonce":2791828047}]},"public_addr":"172.21.15.100:6804/2791828047","cluster_addr":"172.21.15.100:6805/2791828047","heartbeat_back_addr":"172.21.15.100:6807/2791828047","heartbeat_front_addr":"172.21.15.100:6806/2791828047","state":["exists","up"]},{"osd":4,"uuid":"08a1700b-63c5-4028-824f-dbd38931ddb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6808","nonce":4242749853}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6809","nonce":4242749853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6811","nonce":4242749853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6810","nonce":4242749853}]},"public_addr":"172.21.15.100:6808/4242749853","cluster_addr":"172.21.15.100:6809/4242749853","heartbeat_back_addr":"172.21.15.100:6811/4242749853","heartbeat_front_addr":"172.21.15.100:6810/4242749853","state":["exists","up"]},{"osd":5,"uuid":"81fa403f-39e9-44fd-8b1f-ef72fd7ba855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6800","nonce":4145568194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6801","nonce":4145568194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6803","nonce":4145568194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6802","nonce":4145568194}]},"public_addr":"172.21.15.100:6800/4145568194","cluster_addr":"172.21.15.100:6801/4145568194","heartbeat_back_addr":"172.21.15.100:6803/4145568194","heartbeat_front_addr":"172.21.15.100:6802/4145568194","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.703397+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.212068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.432455+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.332164+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.533990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.237115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:46:00.084 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.082+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3055225924 >> v1:172.21.15.100:6812/4210963484 conn(0x7f1794040ce0 legacy=0x7f17940431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:00.084 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.082+0000 7f17b7ff4640 10 monclient: shutdown 2024-09-17T15:46:00.085 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.082+0000 7f17b7ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:00.085 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.082+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3055225924 >> v1:172.21.15.64:6791/0 conn(0x7f17b00a5b10 legacy=0x7f17b01d2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:00.085 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.082+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3055225924 shutdown_connections 2024-09-17T15:46:00.085 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.082+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3055225924 >> 172.21.15.64:0/3055225924 conn(0x7f17b00a8030 msgr2=0x7f17b0142510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:00.085 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.082+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3055225924 shutdown_connections 2024-09-17T15:46:00.085 INFO:teuthology.misc.health.smithi064.stderr:2024-09-17T15:46:00.082+0000 7f17b7ff4640 1 -- 172.21.15.64:0/3055225924 wait complete. 2024-09-17T15:46:00.093 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-17T15:46:00.094 INFO:tasks.ceph:Creating RBD pool 2024-09-17T15:46:00.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-17T15:46:00.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.193+0000 7f742bf89640 10 monclient: get_monmap_and_config 2024-09-17T15:46:00.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.193+0000 7f742bf89640 10 monclient: build_initial_monmap 2024-09-17T15:46:00.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.193+0000 7f742bf89640 10 monclient: monmap: 2024-09-17T15:46:00.195 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:00.195 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:00.195 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:00.194571+0000 2024-09-17T15:46:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:00.194571+0000 2024-09-17T15:46:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:00.196 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:00.196 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:00.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.194+0000 7f742bf89640 1 Processor -- start 2024-09-17T15:46:00.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.194+0000 7f742bf89640 1 -- start start 2024-09-17T15:46:00.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.194+0000 7f742bf89640 10 monclient: init 2024-09-17T15:46:00.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.194+0000 7f742bf89640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:00.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.194+0000 7f742bf89640 10 monclient: _add_conns ranks=[8,1,3,5,6,0,2,4,7] 2024-09-17T15:46:00.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.194+0000 7f742bf89640 10 monclient(hunting): picked mon.noname-e con 0x7f74240a5600 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:00.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f742bf89640 10 monclient(hunting): picked mon.noname-f con 0x7f74241397e0 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:00.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f742bf89640 10 monclient(hunting): picked mon.noname-g con 0x7f742413f800 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:00.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f742bf89640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f742409dab0 con 0x7f74241397e0 2024-09-17T15:46:00.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f742bf89640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f742409dd90 con 0x7f742413f800 2024-09-17T15:46:00.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f742bf89640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f742409e070 con 0x7f74240a5600 2024-09-17T15:46:00.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f742bf89640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:00.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f742bf89640 10 monclient(hunting): authenticate will time out at 757.870067s 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7429d25640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f74240a5600 0x7f74240a59e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:39816/0 (socket says 172.21.15.64:39816) 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7429d25640 1 -- 172.21.15.64:0/2532550037 learned_addr learned my addr 172.21.15.64:0/2532550037 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 1 -- 172.21.15.64:0/2532550037 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2553960805 0 0) 0x7f742409e070 con 0x7f74240a5600 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient(hunting): _negotiate 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient(hunting): my global_id is 4211 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 1 -- 172.21.15.64:0/2532550037 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7410003520 con 0x7f74240a5600 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 1 -- 172.21.15.64:0/2532550037 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4147342742 0 0) 0x7f742409dd90 con 0x7f742413f800 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient(hunting): _negotiate 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient(hunting): my global_id is 4251 2024-09-17T15:46:00.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 1 -- 172.21.15.64:0/2532550037 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f742409e070 con 0x7f742413f800 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 1 -- 172.21.15.64:0/2532550037 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 556137602 0 0) 0x7f7410003520 con 0x7f74240a5600 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 1 -- 172.21.15.64:0/2532550037 >> v1:172.21.15.100:6790/0 conn(0x7f742413f800 legacy=0x7f7424141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 1 -- 172.21.15.64:0/2532550037 >> v1:172.21.15.100:6789/0 conn(0x7f74241397e0 legacy=0x7f742413e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient: _finish_hunting 0 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 1 monclient: found mon.noname-e 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 1 -- 172.21.15.64:0/2532550037 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742409ca00 con 0x7f74240a5600 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient: _finish_auth 0 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient: _check_auth_tickets 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f742bf89640 5 monclient: authenticate success, global_id 4211 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 1 -- 172.21.15.64:0/2532550037 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7420001830 con 0x7f74240a5600 2024-09-17T15:46:00.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f742bf89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient: dump: 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:00.207 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f742bf89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 1 -- 172.21.15.64:0/2532550037 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f74200019d0 con 0x7f74240a5600 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f742bf89640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f742bf89640 4 set_mon_vals no callback set 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 1 -- 172.21.15.64:0/2532550037 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7420004be0 con 0x7f74240a5600 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:00.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.195+0000 7f7428d23640 10 monclient: dump: 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.196+0000 7f742bf89640 10 monclient: shutdown 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.196+0000 7f742bf89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.196+0000 7f742bf89640 1 -- 172.21.15.64:0/2532550037 >> v1:172.21.15.64:6793/0 conn(0x7f74240a5600 legacy=0x7f74240a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.196+0000 7f742bf89640 1 -- 172.21.15.64:0/2532550037 shutdown_connections 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.196+0000 7f742bf89640 1 -- 172.21.15.64:0/2532550037 >> 172.21.15.64:0/2532550037 conn(0x7f742406dd40 msgr2=0x7f74240aba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.196+0000 7f742bf89640 1 -- 172.21.15.64:0/2532550037 shutdown_connections 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.196+0000 7f742bf89640 1 -- 172.21.15.64:0/2532550037 wait complete. 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.196+0000 7f742bf89640 10 monclient: build_initial_monmap 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 10 monclient: monmap: 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:00.212 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:00.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 1 Processor -- start 2024-09-17T15:46:00.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 1 -- start start 2024-09-17T15:46:00.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 10 monclient: init 2024-09-17T15:46:00.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:00.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 10 monclient: _add_conns ranks=[2,8,0,4,1,6,7,5,3] 2024-09-17T15:46:00.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 10 monclient(hunting): picked mon.noname-b con 0x7f74240a5600 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:00.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 10 monclient(hunting): picked mon.noname-e con 0x7f74241397e0 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:00.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 10 monclient(hunting): picked mon.noname-a con 0x7f742413f800 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:00.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f742409f0d0 con 0x7f742413f800 2024-09-17T15:46:00.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74241387d0 con 0x7f74240a5600 2024-09-17T15:46:00.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74240aadd0 con 0x7f74241397e0 2024-09-17T15:46:00.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:00.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.197+0000 7f742bf89640 10 monclient(hunting): authenticate will time out at 757.872961s 2024-09-17T15:46:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f7429524640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f74241397e0 0x7f74241d2d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:39824/0 (socket says 172.21.15.64:39824) 2024-09-17T15:46:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f7429d25640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f74240a5600 0x7f742413c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:48172/0 (socket says 172.21.15.64:48172) 2024-09-17T15:46:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f7429524640 1 -- 172.21.15.64:0/748072683 learned_addr learned my addr 172.21.15.64:0/748072683 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f742a526640 1 --1- 172.21.15.64:0/748072683 >> v1:172.21.15.64:6789/0 conn(0x7f742413f800 0x7f74241d6480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:38786/0 (socket says 172.21.15.64:38786) 2024-09-17T15:46:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1235628845 0 0) 0x7f74241387d0 con 0x7f74240a5600 2024-09-17T15:46:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): my global_id is 4196 2024-09-17T15:46:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7408003520 con 0x7f74240a5600 2024-09-17T15:46:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1899824506 0 0) 0x7f742409f0d0 con 0x7f742413f800 2024-09-17T15:46:00.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:00.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:00.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:00.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): my global_id is 4221 2024-09-17T15:46:00.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74241387d0 con 0x7f742413f800 2024-09-17T15:46:00.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2734079533 0 0) 0x7f74240aadd0 con 0x7f74241397e0 2024-09-17T15:46:00.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:00.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): my global_id is 4220 2024-09-17T15:46:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f742409f0d0 con 0x7f74241397e0 2024-09-17T15:46:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3210758737 0 0) 0x7f74241387d0 con 0x7f742413f800 2024-09-17T15:46:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74240aadd0 con 0x7f742413f800 2024-09-17T15:46:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f741c001db0 con 0x7f742413f800 2024-09-17T15:46:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): dump: 2024-09-17T15:46:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:00.217 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:00.217 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:00.217 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:00.217 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:00.217 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:00.217 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:00.217 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:00.217 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:00.217 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:00.218 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 300731461 0 0) 0x7f7408003520 con 0x7f74240a5600 2024-09-17T15:46:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74241387d0 con 0x7f74240a5600 2024-09-17T15:46:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2718688321 0 0) 0x7f742409f0d0 con 0x7f74241397e0 2024-09-17T15:46:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7408003520 con 0x7f74241397e0 2024-09-17T15:46:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7420004b20 con 0x7f74240a5600 2024-09-17T15:46:00.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:00.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:00.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): dump: 2024-09-17T15:46:00.219 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:00.219 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:00.219 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:00.219 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:00.219 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:00.219 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:00.220 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:00.220 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:00.220 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:00.221 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7414001ad0 con 0x7f74241397e0 2024-09-17T15:46:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:00.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:00.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient(hunting): dump: 2024-09-17T15:46:00.222 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:00.222 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:00.222 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:00.222 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4195185171 0 0) 0x7f74240aadd0 con 0x7f742413f800 2024-09-17T15:46:00.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 >> v1:172.21.15.64:6793/0 conn(0x7f74241397e0 legacy=0x7f74241d2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:00.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 >> v1:172.21.15.64:6790/0 conn(0x7f74240a5600 legacy=0x7f742413c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:00.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:00.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 monclient: found mon.a 2024-09-17T15:46:00.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:00.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:00.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74240aa8a0 con 0x7f742413f800 2024-09-17T15:46:00.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:00.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:00.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f741a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:00.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f742bf89640 5 monclient: authenticate success, global_id 4221 2024-09-17T15:46:00.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f742bf89640 10 monclient: _renew_subs 2024-09-17T15:46:00.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f742bf89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:00.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f742bf89640 1 -- 172.21.15.64:0/748072683 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f742409edf0 con 0x7f742413f800 2024-09-17T15:46:00.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f742bf89640 10 monclient: _renew_subs 2024-09-17T15:46:00.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f742bf89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:00.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.198+0000 7f742bf89640 1 -- 172.21.15.64:0/748072683 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f742413cc00 con 0x7f742413f800 2024-09-17T15:46:00.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.199+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f741c002050 con 0x7f742413f800 2024-09-17T15:46:00.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.199+0000 7f741a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:00.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.199+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f741c003f80 con 0x7f742413f800 2024-09-17T15:46:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.199+0000 7f741a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.199+0000 7f741bfff640 4 set_mon_vals no callback set 2024-09-17T15:46:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.199+0000 7f741a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.199+0000 7f741a7fc640 10 monclient: dump: 2024-09-17T15:46:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:00.227 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:00.227 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:00.227 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:00.227 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:00.227 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:00.227 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:00.227 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:00.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.199+0000 7f742bf89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:00.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.199+0000 7f742bf89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:00.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.199+0000 7f742bf89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:00.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.199+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f741c004120 con 0x7f742413f800 2024-09-17T15:46:00.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.199+0000 7f742bf89640 1 -- 172.21.15.64:0/748072683 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f740c005080 con 0x7f742413f800 2024-09-17T15:46:00.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.200+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (unknown 3963117770 0 0) 0x7f741c04eb80 con 0x7f742413f800 2024-09-17T15:46:00.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.200+0000 7f7429d25640 10 monclient: get_auth_request con 0x7f7408040d50 auth_method 0 2024-09-17T15:46:00.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.205+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f741c017340 con 0x7f742413f800 2024-09-17T15:46:00.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.205+0000 7f741a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:00.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.205+0000 7f741a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:00.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.388+0000 7f742bf89640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-17T15:46:00.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.388+0000 7f742bf89640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-17T15:46:00.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.388+0000 7f742bf89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:00.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.388+0000 7f742bf89640 1 -- 172.21.15.64:0/748072683 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f740c005c40 con 0x7f742413f800 2024-09-17T15:46:00.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.925+0000 7f741a7fc640 1 -- 172.21.15.64:0/748072683 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) ==== 109+0+0 (unknown 231814880 0 0) 0x7f741c01bd10 con 0x7f742413f800 2024-09-17T15:46:00.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.925+0000 7f741a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-17T15:46:00.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.925+0000 7f741a7fc640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-17T15:46:00.927 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'rbd' created 2024-09-17T15:46:00.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.929+0000 7f742bf89640 1 -- 172.21.15.64:0/748072683 >> v1:172.21.15.100:6812/4210963484 conn(0x7f7408040d50 legacy=0x7f7408043210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:00.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.929+0000 7f742bf89640 10 monclient: shutdown 2024-09-17T15:46:00.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.929+0000 7f742bf89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:00.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.929+0000 7f742bf89640 1 -- 172.21.15.64:0/748072683 >> v1:172.21.15.64:6789/0 conn(0x7f742413f800 legacy=0x7f74241d6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:00.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.929+0000 7f742bf89640 1 -- 172.21.15.64:0/748072683 shutdown_connections 2024-09-17T15:46:00.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.930+0000 7f742bf89640 1 -- 172.21.15.64:0/748072683 >> 172.21.15.64:0/748072683 conn(0x7f742406dd40 msgr2=0x7f74241356f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:00.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.930+0000 7f742bf89640 1 -- 172.21.15.64:0/748072683 shutdown_connections 2024-09-17T15:46:00.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:00.930+0000 7f742bf89640 1 -- 172.21.15.64:0/748072683 wait complete. 2024-09-17T15:46:00.946 DEBUG:teuthology.orchestra.run.smithi064:> rbd --cluster ceph pool init rbd 2024-09-17T15:46:01.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.007+0000 7f58a900b140 10 monclient: get_monmap_and_config 2024-09-17T15:46:01.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.007+0000 7f58a900b140 10 monclient: build_initial_monmap 2024-09-17T15:46:01.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.007+0000 7f58a900b140 10 monclient: monmap: 2024-09-17T15:46:01.010 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:01.010 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:01.010 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:01.008734+0000 2024-09-17T15:46:01.010 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:01.008734+0000 2024-09-17T15:46:01.010 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:01.011 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:01.011 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:01.011 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:01.011 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:01.011 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:01.011 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:01.011 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:01.011 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:01.011 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:01.011 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:01.012 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:01.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.008+0000 7f58a900b140 1 Processor -- start 2024-09-17T15:46:01.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.008+0000 7f58a900b140 1 -- start start 2024-09-17T15:46:01.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.008+0000 7f58a900b140 10 monclient: init 2024-09-17T15:46:01.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a900b140 10 monclient: _reopen_session rank -1 2024-09-17T15:46:01.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a900b140 10 monclient: _add_conns ranks=[0,4,8,7,2,3,1,5,6] 2024-09-17T15:46:01.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a900b140 10 monclient(hunting): picked mon.noname-a con 0x55b77cf6abd0 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:01.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a900b140 10 monclient(hunting): picked mon.noname-c con 0x55b77cf6b740 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a900b140 10 monclient(hunting): picked mon.noname-e con 0x55b77d00b170 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a900b140 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55b77cdebd90 con 0x55b77cf6abd0 2024-09-17T15:46:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a900b140 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55b77ce03410 con 0x55b77cf6b740 2024-09-17T15:46:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a900b140 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55b77cdba720 con 0x55b77d00b170 2024-09-17T15:46:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a900b140 10 monclient(hunting): _renew_subs 2024-09-17T15:46:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a900b140 10 monclient(hunting): authenticate will time out at 758.684265s 2024-09-17T15:46:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a8d58640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x55b77d00b170 0x55b77d00d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:39834/0 (socket says 172.21.15.64:39834) 2024-09-17T15:46:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a72f5640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x55b77cf6b740 0x55b77d009a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:50600/0 (socket says 172.21.15.64:50600) 2024-09-17T15:46:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a7af6640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x55b77cf6abd0 0x55b77cf6afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:38788/0 (socket says 172.21.15.64:38788) 2024-09-17T15:46:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a8d58640 1 -- 172.21.15.64:0/1992515045 learned_addr learned my addr 172.21.15.64:0/1992515045 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 1 -- 172.21.15.64:0/1992515045 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3721197084 0 0) 0x55b77ce03410 con 0x55b77cf6b740 2024-09-17T15:46:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 10 monclient(hunting): _negotiate 2024-09-17T15:46:01.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 10 monclient(hunting): my global_id is 4279 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 1 -- 172.21.15.64:0/1992515045 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5894001e10 con 0x55b77cf6b740 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 1 -- 172.21.15.64:0/1992515045 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1010684989 0 0) 0x55b77cdebd90 con 0x55b77cf6abd0 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 10 monclient(hunting): _negotiate 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 10 monclient(hunting): my global_id is 4230 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 1 -- 172.21.15.64:0/1992515045 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x55b77ce03410 con 0x55b77cf6abd0 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 1 -- 172.21.15.64:0/1992515045 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1339579294 0 0) 0x55b77cdba720 con 0x55b77d00b170 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 10 monclient(hunting): _negotiate 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:01.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 10 monclient(hunting): my global_id is 4229 2024-09-17T15:46:01.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.009+0000 7f58a6af4640 1 -- 172.21.15.64:0/1992515045 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x55b77cdebd90 con 0x55b77d00b170 2024-09-17T15:46:01.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 1 -- 172.21.15.64:0/1992515045 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3135764583 0 0) 0x55b77cdebd90 con 0x55b77d00b170 2024-09-17T15:46:01.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 1 -- 172.21.15.64:0/1992515045 >> v1:172.21.15.64:6791/0 conn(0x55b77cf6b740 legacy=0x55b77d009a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:01.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 1 -- 172.21.15.64:0/1992515045 >> v1:172.21.15.64:6789/0 conn(0x55b77cf6abd0 legacy=0x55b77cf6afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:01.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 10 monclient: _finish_hunting 0 2024-09-17T15:46:01.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 1 monclient: found mon.noname-e 2024-09-17T15:46:01.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:01.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:46:01.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 1 -- 172.21.15.64:0/1992515045 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b77cf72520 con 0x55b77d00b170 2024-09-17T15:46:01.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 10 monclient: _finish_auth 0 2024-09-17T15:46:01.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 10 monclient: _check_auth_tickets 2024-09-17T15:46:01.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 1 -- 172.21.15.64:0/1992515045 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5898001a50 con 0x55b77d00b170 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 10 monclient: dump: 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:01.024 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a900b140 5 monclient: authenticate success, global_id 4229 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a900b140 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 1 -- 172.21.15.64:0/1992515045 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5898001c70 con 0x55b77d00b170 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 1 -- 172.21.15.64:0/1992515045 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5898003850 con 0x55b77d00b170 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a900b140 10 monclient: get_monmap_and_config success 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a900b140 4 set_mon_vals no callback set 2024-09-17T15:46:01.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:01.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:01.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a6af4640 10 monclient: dump: 2024-09-17T15:46:01.026 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:01.026 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:01.026 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:01.026 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:01.026 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:01.026 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:01.026 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:01.027 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:01.027 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:01.027 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:01.027 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:01.027 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:01.027 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:01.027 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:01.027 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:01.027 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:01.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a900b140 10 monclient: shutdown 2024-09-17T15:46:01.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a900b140 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:01.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a900b140 1 -- 172.21.15.64:0/1992515045 >> v1:172.21.15.64:6793/0 conn(0x55b77d00b170 legacy=0x55b77d00d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:01.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a900b140 1 -- 172.21.15.64:0/1992515045 shutdown_connections 2024-09-17T15:46:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a900b140 1 -- 172.21.15.64:0/1992515045 >> 172.21.15.64:0/1992515045 conn(0x55b77cf755b0 msgr2=0x55b77cf71690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a900b140 1 -- 172.21.15.64:0/1992515045 shutdown_connections 2024-09-17T15:46:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.010+0000 7f58a900b140 1 -- 172.21.15.64:0/1992515045 wait complete. 2024-09-17T15:46:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.011+0000 7f58a900b140 10 monclient: build_initial_monmap 2024-09-17T15:46:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.011+0000 7f58a900b140 10 monclient: monmap: 2024-09-17T15:46:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:01.029 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.011+0000 7f58a900b140 1 Processor -- start 2024-09-17T15:46:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.011+0000 7f58a900b140 1 -- start start 2024-09-17T15:46:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.011+0000 7f58a900b140 10 monclient: init 2024-09-17T15:46:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f58a900b140 10 monclient: _reopen_session rank -1 2024-09-17T15:46:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f58a900b140 10 monclient: _add_conns ranks=[2,3,4,0,8,1,7,6,5] 2024-09-17T15:46:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f58a900b140 10 monclient(hunting): picked mon.noname-b con 0x55b77cf6abd0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f58a900b140 10 monclient(hunting): picked mon.noname-g con 0x55b77cf6b740 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f58a900b140 10 monclient(hunting): picked mon.noname-c con 0x55b77d00b170 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f58a900b140 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55b77d006d50 con 0x55b77cf6abd0 2024-09-17T15:46:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f58a900b140 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55b77d00a6f0 con 0x55b77cf6b740 2024-09-17T15:46:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f58a900b140 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55b77d01e650 con 0x55b77d00b170 2024-09-17T15:46:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f58a900b140 10 monclient(hunting): _renew_subs 2024-09-17T15:46:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f58a900b140 10 monclient(hunting): authenticate will time out at 758.687454s 2024-09-17T15:46:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f58a7af6640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x55b77cf6abd0 0x55b77d003fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:48182/0 (socket says 172.21.15.64:48182) 2024-09-17T15:46:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f58a7af6640 1 -- 172.21.15.64:0/4211858755 learned_addr learned my addr 172.21.15.64:0/4211858755 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f58a8d58640 1 --1- 172.21.15.64:0/4211858755 >> v1:172.21.15.64:6791/0 conn(0x55b77d00b170 0x55b77d004dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:50606/0 (socket says 172.21.15.64:50606) 2024-09-17T15:46:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 538863417 0 0) 0x55b77d01e650 con 0x55b77d00b170 2024-09-17T15:46:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:01.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:01.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:01.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 10 monclient(hunting): my global_id is 4288 2024-09-17T15:46:01.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5888001e10 con 0x55b77d00b170 2024-09-17T15:46:01.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1704445216 0 0) 0x55b77d006d50 con 0x55b77cf6abd0 2024-09-17T15:46:01.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:01.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:01.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:01.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 10 monclient(hunting): my global_id is 4205 2024-09-17T15:46:01.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x55b77d01e650 con 0x55b77cf6abd0 2024-09-17T15:46:01.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2618192327 0 0) 0x55b77d00a6f0 con 0x55b77cf6b740 2024-09-17T15:46:01.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:01.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:01.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.012+0000 7f588ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:01.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient(hunting): my global_id is 4260 2024-09-17T15:46:01.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x55b77d006d50 con 0x55b77cf6b740 2024-09-17T15:46:01.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3414509253 0 0) 0x7f5888001e10 con 0x55b77d00b170 2024-09-17T15:46:01.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x55b77d00a6f0 con 0x55b77d00b170 2024-09-17T15:46:01.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5898001a50 con 0x55b77d00b170 2024-09-17T15:46:01.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:01.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:01.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient(hunting): dump: 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:01.033 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:01.034 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:01.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:01.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3378041495 0 0) 0x55b77d01e650 con 0x55b77cf6abd0 2024-09-17T15:46:01.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5888001e10 con 0x55b77cf6abd0 2024-09-17T15:46:01.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f589c001430 con 0x55b77cf6abd0 2024-09-17T15:46:01.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:01.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:01.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient(hunting): dump: 2024-09-17T15:46:01.034 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:01.034 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:01.034 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:01.034 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:01.034 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:01.035 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:01.035 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:01.035 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:01.036 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:01.036 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:01.036 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:01.036 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:01.036 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:01.036 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:01.036 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:01.036 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:01.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3778492819 0 0) 0x55b77d006d50 con 0x55b77cf6b740 2024-09-17T15:46:01.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x55b77d01e650 con 0x55b77cf6b740 2024-09-17T15:46:01.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5890001590 con 0x55b77cf6b740 2024-09-17T15:46:01.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:46:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient(hunting): dump: 2024-09-17T15:46:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:01.037 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:01.038 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:01.038 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:01.038 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:01.038 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:01.038 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:01.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 92754733 0 0) 0x55b77d00a6f0 con 0x55b77d00b170 2024-09-17T15:46:01.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 >> v1:172.21.15.100:6790/0 conn(0x55b77cf6b740 legacy=0x55b77d0046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:01.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 >> v1:172.21.15.64:6790/0 conn(0x55b77cf6abd0 legacy=0x55b77d003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:01.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:01.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 monclient: found mon.c 2024-09-17T15:46:01.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:01.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:01.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b77d01f830 con 0x55b77d00b170 2024-09-17T15:46:01.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient: _finish_auth 0 2024-09-17T15:46:01.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:01.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:01.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f58a900b140 5 monclient: authenticate success, global_id 4288 2024-09-17T15:46:01.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f58a900b140 10 monclient: _renew_subs 2024-09-17T15:46:01.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f58a900b140 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:01.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x55b77d01e820 con 0x55b77d00b170 2024-09-17T15:46:01.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f58a900b140 10 monclient: _renew_subs 2024-09-17T15:46:01.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f58a900b140 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:01.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x55b77d01ede0 con 0x55b77d00b170 2024-09-17T15:46:01.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5898003700 con 0x55b77d00b170 2024-09-17T15:46:01.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:01.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5898003c30 con 0x55b77d00b170 2024-09-17T15:46:01.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:01.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:01.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f588ffff640 10 monclient: dump: 2024-09-17T15:46:01.040 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:01.040 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:01.041 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:01.041 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:01.041 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:01.041 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:01.041 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:01.041 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:01.041 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:01.041 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:01.041 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:01.041 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:01.041 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:01.041 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:01.041 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:01.042 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:01.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.013+0000 7f58a62f3640 4 set_mon_vals no callback set 2024-09-17T15:46:01.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.014+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f5898003dd0 con 0x55b77d00b170 2024-09-17T15:46:01.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.014+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(16..16 src has 1..16) ==== 3495+0+0 (unknown 743456970 0 0) 0x7f589804d7e0 con 0x55b77d00b170 2024-09-17T15:46:01.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.014+0000 7f58a900b140 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-17T15:46:01.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.014+0000 7f58a900b140 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-17T15:46:01.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.014+0000 7f58a900b140 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:01.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.014+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x55b77d01f330 con 0x55b77d00b170 2024-09-17T15:46:01.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.014+0000 7f58a7af6640 10 monclient: get_auth_request con 0x7f588803f610 auth_method 0 2024-09-17T15:46:01.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.935+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v17) ==== 141+0+0 (unknown 1851909647 0 0) 0x7f5898016560 con 0x55b77d00b170 2024-09-17T15:46:01.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.935+0000 7f588ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-17T15:46:01.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.935+0000 7f588ffff640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-17T15:46:01.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.935+0000 7f58a900b140 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:01.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.936+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6791/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5898001c70 con 0x55b77d00b170 2024-09-17T15:46:01.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.936+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 10 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7f5898015810 con 0x55b77d00b170 2024-09-17T15:46:01.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.936+0000 7f588ffff640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-17T15:46:01.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.937+0000 7f58a62f3640 10 monclient: _renew_subs 2024-09-17T15:46:01.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.937+0000 7f58a62f3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:01.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.937+0000 7f58a62f3640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=17}) -- 0x7f5880002bc0 con 0x55b77d00b170 2024-09-17T15:46:01.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.937+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 11 ==== osd_map(17..17 src has 1..17) ==== 658+0+0 (unknown 3210334808 0 0) 0x7f589804c6c0 con 0x55b77d00b170 2024-09-17T15:46:01.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.937+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.100:6800/4145568194 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x55b77d01dae0 con 0x55b77d01cff0 2024-09-17T15:46:01.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.938+0000 7f58a72f5640 10 monclient: get_auth_request con 0x55b77d01cff0 auth_method 0 2024-09-17T15:46:01.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.940+0000 7f58a72f5640 1 -- 172.21.15.64:0/4211858755 <== osd.5 v1:172.21.15.100:6800/4145568194 1 ==== osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) ==== 153+0+0 (unknown 1703302865 0 0) 0x7f5890004240 con 0x55b77d01cff0 2024-09-17T15:46:01.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.943+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.100:6800/4145568194 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55b77d073940 con 0x55b77d01cff0 2024-09-17T15:46:01.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.943+0000 7f58a72f5640 1 -- 172.21.15.64:0/4211858755 <== osd.5 v1:172.21.15.100:6800/4145568194 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 4274215525 0 0) 0x7f5890004240 con 0x55b77d01cff0 2024-09-17T15:46:01.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.943+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.100:6800/4145568194 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55b77d072a50 con 0x55b77d01cff0 2024-09-17T15:46:01.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.943+0000 7f58a72f5640 1 -- 172.21.15.64:0/4211858755 <== osd.5 v1:172.21.15.100:6800/4145568194 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 4287864937 0 0) 0x7f5890004240 con 0x55b77d01cff0 2024-09-17T15:46:01.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.943+0000 7f58a5af2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:01.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:01.943+0000 7f58a5af2640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6791/0 -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f5884001f00 con 0x55b77d00b170 2024-09-17T15:46:02.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:02.937+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 12 ==== pool_op_reply(tid 4 (0) Success v18) ==== 55+0+0 (unknown 1071121186 0 0) 0x7f5898015e60 con 0x55b77d00b170 2024-09-17T15:46:02.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:02.937+0000 7f588ffff640 10 monclient: _renew_subs 2024-09-17T15:46:02.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:02.937+0000 7f588ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:02.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:02.937+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=18}) -- 0x55b77d00a6f0 con 0x55b77d00b170 2024-09-17T15:46:02.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:02.938+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 13 ==== osd_map(18..18 src has 1..18) ==== 626+0+0 (unknown 2031436208 0 0) 0x7f589801f2d0 con 0x55b77d00b170 2024-09-17T15:46:02.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:02.938+0000 7f58a62f3640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.100:6800/4145568194 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7f58800058d0 con 0x55b77d01cff0 2024-09-17T15:46:02.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:02.940+0000 7f58a72f5640 1 -- 172.21.15.64:0/4211858755 <== osd.5 v1:172.21.15.100:6800/4145568194 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) ==== 194+0+0 (unknown 2910594948 0 0) 0x7f5890004240 con 0x55b77d01cff0 2024-09-17T15:46:02.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:02.940+0000 7f58a5af2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:02.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:02.940+0000 7f58a5af2640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6791/0 -- pool_op(delete unmanaged snap pool 2 tid 6 name v18) -- 0x7f5884003370 con 0x55b77d00b170 2024-09-17T15:46:03.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.955+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 14 ==== pool_op_reply(tid 6 (0) Success v19) ==== 43+0+0 (unknown 2054842491 0 0) 0x7f589801f760 con 0x55b77d00b170 2024-09-17T15:46:03.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.955+0000 7f588ffff640 10 monclient: _renew_subs 2024-09-17T15:46:03.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.955+0000 7f588ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:03.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.955+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=19}) -- 0x55b77d006d50 con 0x55b77d00b170 2024-09-17T15:46:03.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.955+0000 7f588ffff640 1 -- 172.21.15.64:0/4211858755 <== mon.4 v1:172.21.15.64:6791/0 15 ==== osd_map(19..19 src has 1..19) ==== 654+0+0 (unknown 2227655083 0 0) 0x7f589801f4d0 con 0x55b77d00b170 2024-09-17T15:46:03.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.955+0000 7f58a62f3640 1 -- 172.21.15.64:0/4211858755 --> v1:172.21.15.100:6800/4145568194 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x7f5880005c70 con 0x55b77d01cff0 2024-09-17T15:46:03.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.957+0000 7f58a72f5640 1 -- 172.21.15.64:0/4211858755 <== osd.5 v1:172.21.15.100:6800/4145568194 5 ==== osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) ==== 152+0+0 (unknown 900577457 0 0) 0x7f5890004240 con 0x55b77d01cff0 2024-09-17T15:46:03.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.957+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 >> v1:172.21.15.100:6800/4145568194 conn(0x55b77d01cff0 legacy=0x55b77d01d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:03.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.957+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 >> v1:172.21.15.100:6812/4210963484 conn(0x7f588803f610 legacy=0x7f5888041ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:03.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.957+0000 7f58a900b140 10 monclient: shutdown 2024-09-17T15:46:03.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.957+0000 7f58a900b140 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:03.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.957+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 >> v1:172.21.15.64:6791/0 conn(0x55b77d00b170 legacy=0x55b77d004dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:03.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.957+0000 7f58a8d58640 1 -- 172.21.15.64:0/4211858755 reap_dead start 2024-09-17T15:46:03.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.958+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 shutdown_connections 2024-09-17T15:46:03.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.958+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 >> 172.21.15.64:0/4211858755 conn(0x55b77cf755b0 msgr2=0x55b77d0016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:03.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.958+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 shutdown_connections 2024-09-17T15:46:03.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:03.958+0000 7f58a900b140 1 -- 172.21.15.64:0/4211858755 wait complete. 2024-09-17T15:46:03.963 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-17T15:46:03.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-17T15:46:03.964 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-17T15:46:04.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.071+0000 7f61eb47f640 10 monclient: get_monmap_and_config 2024-09-17T15:46:04.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.071+0000 7f61eb47f640 10 monclient: build_initial_monmap 2024-09-17T15:46:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.071+0000 7f61eb47f640 10 monclient: monmap: 2024-09-17T15:46:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:04.072466+0000 2024-09-17T15:46:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:04.072466+0000 2024-09-17T15:46:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:04.074 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:04.075 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:04.075 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:04.075 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:04.075 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.072+0000 7f61eb47f640 1 Processor -- start 2024-09-17T15:46:04.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.072+0000 7f61eb47f640 1 -- start start 2024-09-17T15:46:04.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.072+0000 7f61eb47f640 10 monclient: init 2024-09-17T15:46:04.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.072+0000 7f61eb47f640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:04.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.072+0000 7f61eb47f640 10 monclient: _add_conns ranks=[2,6,0,3,1,4,8,7,5] 2024-09-17T15:46:04.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.072+0000 7f61eb47f640 10 monclient(hunting): picked mon.noname-b con 0x7f61e40a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:04.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.072+0000 7f61eb47f640 10 monclient(hunting): picked mon.noname-d con 0x7f61e4135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:04.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.072+0000 7f61eb47f640 10 monclient(hunting): picked mon.noname-a con 0x7f61e4135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:04.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.072+0000 7f61eb47f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61e409d6f0 con 0x7f61e4135f70 2024-09-17T15:46:04.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.072+0000 7f61eb47f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61e409d9d0 con 0x7f61e40a5520 2024-09-17T15:46:04.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.072+0000 7f61eb47f640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61e409dcb0 con 0x7f61e4135b50 2024-09-17T15:46:04.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.072+0000 7f61eb47f640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:04.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.072+0000 7f61eb47f640 10 monclient(hunting): authenticate will time out at 761.747992s 2024-09-17T15:46:04.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61e921b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f61e40a5520 0x7f61e40a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:48186/0 (socket says 172.21.15.64:48186) 2024-09-17T15:46:04.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61e8a1a640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f61e4135b50 0x7f61e413deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:38272/0 (socket says 172.21.15.64:38272) 2024-09-17T15:46:04.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61e9a1c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f61e4135f70 0x7f61e41415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:38808/0 (socket says 172.21.15.64:38808) 2024-09-17T15:46:04.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61e921b640 1 -- 172.21.15.64:0/1179680730 learned_addr learned my addr 172.21.15.64:0/1179680730 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:04.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 -- 172.21.15.64:0/1179680730 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1710819242 0 0) 0x7f61e409d9d0 con 0x7f61e40a5520 2024-09-17T15:46:04.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient(hunting): my global_id is 4214 2024-09-17T15:46:04.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 -- 172.21.15.64:0/1179680730 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61d0003520 con 0x7f61e40a5520 2024-09-17T15:46:04.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 -- 172.21.15.64:0/1179680730 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 474635317 0 0) 0x7f61e409d6f0 con 0x7f61e4135f70 2024-09-17T15:46:04.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient(hunting): my global_id is 4239 2024-09-17T15:46:04.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 -- 172.21.15.64:0/1179680730 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61e409d9d0 con 0x7f61e4135f70 2024-09-17T15:46:04.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 -- 172.21.15.64:0/1179680730 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 703186195 0 0) 0x7f61e409dcb0 con 0x7f61e4135b50 2024-09-17T15:46:04.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient(hunting): my global_id is 4227 2024-09-17T15:46:04.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 -- 172.21.15.64:0/1179680730 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61e409d6f0 con 0x7f61e4135b50 2024-09-17T15:46:04.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 -- 172.21.15.64:0/1179680730 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1227453699 0 0) 0x7f61d0003520 con 0x7f61e40a5520 2024-09-17T15:46:04.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 -- 172.21.15.64:0/1179680730 >> v1:172.21.15.64:6792/0 conn(0x7f61e4135b50 legacy=0x7f61e413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 -- 172.21.15.64:0/1179680730 >> v1:172.21.15.64:6789/0 conn(0x7f61e4135f70 legacy=0x7f61e41415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:04.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 monclient: found mon.noname-b 2024-09-17T15:46:04.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:04.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:46:04.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 -- 172.21.15.64:0/1179680730 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e409ccd0 con 0x7f61e40a5520 2024-09-17T15:46:04.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient: _finish_auth 0 2024-09-17T15:46:04.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:04.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:04.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 -- 172.21.15.64:0/1179680730 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f61e0001830 con 0x7f61e40a5520 2024-09-17T15:46:04.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:04.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:04.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient: dump: 2024-09-17T15:46:04.087 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.087 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.087 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.087 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.087 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.087 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.087 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.087 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.087 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.087 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.087 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.087 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.088 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.088 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.088 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.088 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61eb47f640 5 monclient: authenticate success, global_id 4214 2024-09-17T15:46:04.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61eb47f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:04.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 -- 172.21.15.64:0/1179680730 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f61e0001a50 con 0x7f61e40a5520 2024-09-17T15:46:04.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:04.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 1 -- 172.21.15.64:0/1179680730 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f61e0001d40 con 0x7f61e40a5520 2024-09-17T15:46:04.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61eb47f640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:04.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61eb47f640 4 set_mon_vals no callback set 2024-09-17T15:46:04.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:04.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:04.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.073+0000 7f61dbfff640 10 monclient: dump: 2024-09-17T15:46:04.089 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.089 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.089 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.089 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.089 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.089 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.089 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.089 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.089 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.089 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.089 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.090 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.090 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.090 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.090 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.090 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.074+0000 7f61eb47f640 10 monclient: shutdown 2024-09-17T15:46:04.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.074+0000 7f61eb47f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:04.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.074+0000 7f61eb47f640 1 -- 172.21.15.64:0/1179680730 >> v1:172.21.15.64:6790/0 conn(0x7f61e40a5520 legacy=0x7f61e40a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.074+0000 7f61eb47f640 1 -- 172.21.15.64:0/1179680730 shutdown_connections 2024-09-17T15:46:04.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.074+0000 7f61eb47f640 1 -- 172.21.15.64:0/1179680730 >> 172.21.15.64:0/1179680730 conn(0x7f61e406db30 msgr2=0x7f61e40ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:04.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.074+0000 7f61eb47f640 1 -- 172.21.15.64:0/1179680730 shutdown_connections 2024-09-17T15:46:04.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.074+0000 7f61eb47f640 1 -- 172.21.15.64:0/1179680730 wait complete. 2024-09-17T15:46:04.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 10 monclient: build_initial_monmap 2024-09-17T15:46:04.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 10 monclient: monmap: 2024-09-17T15:46:04.091 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:04.091 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:04.091 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:04.091 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:04.091 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:04.091 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:04.091 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:04.092 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:04.092 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:04.092 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:04.092 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:04.092 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:04.092 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:04.092 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:04.092 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:04.093 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 1 Processor -- start 2024-09-17T15:46:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 1 -- start start 2024-09-17T15:46:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 10 monclient: init 2024-09-17T15:46:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 10 monclient: _add_conns ranks=[8,6,7,1,5,2,3,4,0] 2024-09-17T15:46:04.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 10 monclient(hunting): picked mon.noname-e con 0x7f61e40a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:04.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 10 monclient(hunting): picked mon.noname-d con 0x7f61e4135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:04.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 10 monclient(hunting): picked mon.noname-i con 0x7f61e4135f70 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:04.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61e413a1f0 con 0x7f61e4135b50 2024-09-17T15:46:04.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61e40a6710 con 0x7f61e4135f70 2024-09-17T15:46:04.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61e4138730 con 0x7f61e40a5520 2024-09-17T15:46:04.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:04.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.075+0000 7f61eb47f640 10 monclient(hunting): authenticate will time out at 761.750956s 2024-09-17T15:46:04.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61e8a1a640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f61e4135b50 0x7f61e4137390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:38282/0 (socket says 172.21.15.64:38282) 2024-09-17T15:46:04.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61e921b640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f61e40a5520 0x7f61e4136c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:39860/0 (socket says 172.21.15.64:39860) 2024-09-17T15:46:04.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61e8a1a640 1 -- 172.21.15.64:0/1219150865 learned_addr learned my addr 172.21.15.64:0/1219150865 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:04.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1643951192 0 0) 0x7f61e4138730 con 0x7f61e40a5520 2024-09-17T15:46:04.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): my global_id is 4238 2024-09-17T15:46:04.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61c8003520 con 0x7f61e40a5520 2024-09-17T15:46:04.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 477701602 0 0) 0x7f61e409d6f0 con 0x7f61e4135b50 2024-09-17T15:46:04.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): my global_id is 4236 2024-09-17T15:46:04.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61e4138730 con 0x7f61e4135b50 2024-09-17T15:46:04.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3435545294 0 0) 0x7f61c8003520 con 0x7f61e40a5520 2024-09-17T15:46:04.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61e409d6f0 con 0x7f61e40a5520 2024-09-17T15:46:04.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f61e00017f0 con 0x7f61e40a5520 2024-09-17T15:46:04.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:04.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:46:04.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): dump: 2024-09-17T15:46:04.096 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.096 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.096 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.096 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.096 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.097 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.097 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.097 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.097 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.097 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.097 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.098 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.098 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.098 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.098 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.098 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:04.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1989615458 0 0) 0x7f61e4138730 con 0x7f61e4135b50 2024-09-17T15:46:04.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61c8003520 con 0x7f61e4135b50 2024-09-17T15:46:04.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f61d4001a90 con 0x7f61e4135b50 2024-09-17T15:46:04.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:04.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): dump: 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.099 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.100 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.100 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 118181702 0 0) 0x7f61e40a6710 con 0x7f61e4135f70 2024-09-17T15:46:04.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient(hunting): my global_id is 4255 2024-09-17T15:46:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61e4138730 con 0x7f61e4135f70 2024-09-17T15:46:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 952314801 0 0) 0x7f61e409d6f0 con 0x7f61e40a5520 2024-09-17T15:46:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 >> v1:172.21.15.100:6792/0 conn(0x7f61e4135f70 legacy=0x7f61e41d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 >> v1:172.21.15.64:6792/0 conn(0x7f61e4135b50 legacy=0x7f61e4137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:46:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 monclient: found mon.e 2024-09-17T15:46:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e4137aa0 con 0x7f61e40a5520 2024-09-17T15:46:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61eb47f640 5 monclient: authenticate success, global_id 4238 2024-09-17T15:46:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61eb47f640 10 monclient: _renew_subs 2024-09-17T15:46:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61eb47f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61eb47f640 1 -- 172.21.15.64:0/1219150865 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f61e4141730 con 0x7f61e40a5520 2024-09-17T15:46:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f61e0004d40 con 0x7f61e40a5520 2024-09-17T15:46:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f61e0005230 con 0x7f61e40a5520 2024-09-17T15:46:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.077+0000 7f61d9ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.077+0000 7f61d9ffb640 10 monclient: dump: 2024-09-17T15:46:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.103 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.103 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.103 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.103 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.103 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.103 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.103 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.103 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.103 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.103 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.103 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.104 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.104 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.104 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.077+0000 7f61eb47f640 10 monclient: _renew_subs 2024-09-17T15:46:04.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.076+0000 7f61db7fe640 4 set_mon_vals no callback set 2024-09-17T15:46:04.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.077+0000 7f61eb47f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:04.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.077+0000 7f61eb47f640 1 -- 172.21.15.64:0/1219150865 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f61e41420e0 con 0x7f61e40a5520 2024-09-17T15:46:04.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.077+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f61e0011b90 con 0x7f61e40a5520 2024-09-17T15:46:04.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.078+0000 7f61eb47f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:04.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.078+0000 7f61eb47f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:04.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.078+0000 7f61eb47f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:04.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.078+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f61e004eb70 con 0x7f61e40a5520 2024-09-17T15:46:04.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.078+0000 7f61eb47f640 1 -- 172.21.15.64:0/1219150865 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61e4142410 con 0x7f61e40a5520 2024-09-17T15:46:04.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.078+0000 7f61e8a1a640 10 monclient: get_auth_request con 0x7f61c8040a10 auth_method 0 2024-09-17T15:46:04.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.083+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.8 v1:172.21.15.64:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f61e0017950 con 0x7f61e40a5520 2024-09-17T15:46:04.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.083+0000 7f61d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:04.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.083+0000 7f61d9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:04.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.272+0000 7f61eb47f640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-17T15:46:04.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.272+0000 7f61eb47f640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-17T15:46:04.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.272+0000 7f61eb47f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:04.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.272+0000 7f61eb47f640 1 -- 172.21.15.64:0/1219150865 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f61e4068b50 con 0x7f61e40a5520 2024-09-17T15:46:04.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.272+0000 7f61d9ffb640 1 -- 172.21.15.64:0/1219150865 <== mon.8 v1:172.21.15.64:6793/0 10 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v5) ==== 86+0+159 (unknown 882900978 0 3032913478) 0x7f61e0017250 con 0x7f61e40a5520 2024-09-17T15:46:04.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.272+0000 7f61d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-17T15:46:04.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.272+0000 7f61d9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:04.274 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:46:04.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.274+0000 7f61eb47f640 1 -- 172.21.15.64:0/1219150865 >> v1:172.21.15.100:6812/4210963484 conn(0x7f61c8040a10 legacy=0x7f61c8042ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.274+0000 7f61eb47f640 10 monclient: shutdown 2024-09-17T15:46:04.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.274+0000 7f61eb47f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:04.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.274+0000 7f61eb47f640 1 -- 172.21.15.64:0/1219150865 >> v1:172.21.15.64:6793/0 conn(0x7f61e40a5520 legacy=0x7f61e4136c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.275+0000 7f61eb47f640 1 -- 172.21.15.64:0/1219150865 shutdown_connections 2024-09-17T15:46:04.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.275+0000 7f61eb47f640 1 -- 172.21.15.64:0/1219150865 >> 172.21.15.64:0/1219150865 conn(0x7f61e406db30 msgr2=0x7f61e4133e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:04.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.275+0000 7f61eb47f640 1 -- 172.21.15.64:0/1219150865 shutdown_connections 2024-09-17T15:46:04.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.275+0000 7f61eb47f640 1 -- 172.21.15.64:0/1219150865 wait complete. 2024-09-17T15:46:04.286 INFO:teuthology.orchestra.run.smithi064.stdout:[{"version":5,"timestamp":"2024-09-17T15:45:55.040228+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"5572.087537"}]}] 2024-09-17T15:46:04.287 INFO:tasks.ceph_manager:config epoch is 5 2024-09-17T15:46:04.287 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T15:46:04.287 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-17T15:46:04.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-17T15:46:04.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.386+0000 7fcc27edd640 10 monclient: get_monmap_and_config 2024-09-17T15:46:04.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.386+0000 7fcc27edd640 10 monclient: build_initial_monmap 2024-09-17T15:46:04.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.386+0000 7fcc27edd640 10 monclient: monmap: 2024-09-17T15:46:04.388 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:04.388 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.388 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:04.387562+0000 2024-09-17T15:46:04.389 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:04.387562+0000 2024-09-17T15:46:04.389 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:04.389 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.389 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:04.389 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:04.389 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:04.389 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:04.389 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:04.389 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:04.389 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:04.389 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:04.389 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:04.390 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.387+0000 7fcc27edd640 1 Processor -- start 2024-09-17T15:46:04.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.387+0000 7fcc27edd640 1 -- start start 2024-09-17T15:46:04.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.387+0000 7fcc27edd640 10 monclient: init 2024-09-17T15:46:04.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.387+0000 7fcc27edd640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:04.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.387+0000 7fcc27edd640 10 monclient: _add_conns ranks=[0,7,5,3,6,1,8,4,2] 2024-09-17T15:46:04.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc27edd640 10 monclient(hunting): picked mon.noname-a con 0x7fcc200a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:04.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc27edd640 10 monclient(hunting): picked mon.noname-i con 0x7fcc20135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:04.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc27edd640 10 monclient(hunting): picked mon.noname-h con 0x7fcc20135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:04.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc27edd640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc2009d6f0 con 0x7fcc200a5520 2024-09-17T15:46:04.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc27edd640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc2009d9d0 con 0x7fcc20135f70 2024-09-17T15:46:04.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc27edd640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc2009dcb0 con 0x7fcc20135b50 2024-09-17T15:46:04.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc27edd640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:04.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc27edd640 10 monclient(hunting): authenticate will time out at 762.063077s 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc25c79640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fcc200a5520 0x7fcc200a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:38818/0 (socket says 172.21.15.64:38818) 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc25c79640 1 -- 172.21.15.64:0/3346406285 learned_addr learned my addr 172.21.15.64:0/3346406285 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 1 -- 172.21.15.64:0/3346406285 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2992133596 0 0) 0x7fcc2009d6f0 con 0x7fcc200a5520 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient(hunting): my global_id is 4248 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 1 -- 172.21.15.64:0/3346406285 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc0c003520 con 0x7fcc200a5520 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 1 -- 172.21.15.64:0/3346406285 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4244410257 0 0) 0x7fcc0c003520 con 0x7fcc200a5520 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 1 -- 172.21.15.64:0/3346406285 >> v1:172.21.15.100:6792/0 conn(0x7fcc20135b50 legacy=0x7fcc2013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 1 -- 172.21.15.64:0/3346406285 >> v1:172.21.15.100:6791/0 conn(0x7fcc20135f70 legacy=0x7fcc201415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient: _finish_hunting 0 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 1 monclient: found mon.noname-a 2024-09-17T15:46:04.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:46:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 1 -- 172.21.15.64:0/3346406285 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2009ccd0 con 0x7fcc200a5520 2024-09-17T15:46:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient: _finish_auth 0 2024-09-17T15:46:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient: _check_auth_tickets 2024-09-17T15:46:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 1 -- 172.21.15.64:0/3346406285 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcc1c001830 con 0x7fcc200a5520 2024-09-17T15:46:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient: dump: 2024-09-17T15:46:04.399 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.400 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.400 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.400 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.400 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.400 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.400 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.400 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.400 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.400 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.400 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.400 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.400 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.400 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.401 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.401 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc27edd640 5 monclient: authenticate success, global_id 4248 2024-09-17T15:46:04.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc27edd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:04.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 1 -- 172.21.15.64:0/3346406285 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc1c0019d0 con 0x7fcc200a5520 2024-09-17T15:46:04.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:04.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 1 -- 172.21.15.64:0/3346406285 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcc1c004be0 con 0x7fcc200a5520 2024-09-17T15:46:04.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:04.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:04.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc24c77640 10 monclient: dump: 2024-09-17T15:46:04.401 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.401 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.401 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.402 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.402 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.402 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.402 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.402 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.402 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.402 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.402 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.402 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.402 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.402 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.402 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.402 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc27edd640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.388+0000 7fcc27edd640 4 set_mon_vals no callback set 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.389+0000 7fcc27edd640 10 monclient: shutdown 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.389+0000 7fcc27edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.389+0000 7fcc27edd640 1 -- 172.21.15.64:0/3346406285 >> v1:172.21.15.64:6789/0 conn(0x7fcc200a5520 legacy=0x7fcc200a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.389+0000 7fcc27edd640 1 -- 172.21.15.64:0/3346406285 shutdown_connections 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.389+0000 7fcc27edd640 1 -- 172.21.15.64:0/3346406285 >> 172.21.15.64:0/3346406285 conn(0x7fcc2006db30 msgr2=0x7fcc200ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.389+0000 7fcc27edd640 1 -- 172.21.15.64:0/3346406285 shutdown_connections 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.389+0000 7fcc27edd640 1 -- 172.21.15.64:0/3346406285 wait complete. 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.389+0000 7fcc27edd640 10 monclient: build_initial_monmap 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.389+0000 7fcc27edd640 10 monclient: monmap: 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:04.403 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc27edd640 1 Processor -- start 2024-09-17T15:46:04.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc27edd640 1 -- start start 2024-09-17T15:46:04.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc27edd640 10 monclient: init 2024-09-17T15:46:04.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc27edd640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:04.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc27edd640 10 monclient: _add_conns ranks=[4,6,7,0,5,1,3,8,2] 2024-09-17T15:46:04.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc27edd640 10 monclient(hunting): picked mon.noname-c con 0x7fcc200a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:04.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc27edd640 10 monclient(hunting): picked mon.noname-d con 0x7fcc20135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:04.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc27edd640 10 monclient(hunting): picked mon.noname-i con 0x7fcc20135f70 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:04.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc27edd640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc201346f0 con 0x7fcc200a5520 2024-09-17T15:46:04.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc27edd640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc20134b00 con 0x7fcc20135b50 2024-09-17T15:46:04.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc27edd640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc200a8ef0 con 0x7fcc20135f70 2024-09-17T15:46:04.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc27edd640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:04.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc27edd640 10 monclient(hunting): authenticate will time out at 762.065812s 2024-09-17T15:46:04.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc25c79640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fcc200a5520 0x7fcc20131d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:50620/0 (socket says 172.21.15.64:50620) 2024-09-17T15:46:04.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc25c79640 1 -- 172.21.15.64:0/1945820389 learned_addr learned my addr 172.21.15.64:0/1945820389 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:04.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.390+0000 7fcc25478640 1 --1- 172.21.15.64:0/1945820389 >> v1:172.21.15.64:6792/0 conn(0x7fcc20135b50 0x7fcc20132450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:38284/0 (socket says 172.21.15.64:38284) 2024-09-17T15:46:04.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3879063300 0 0) 0x7fcc2009ccd0 con 0x7fcc200a5520 2024-09-17T15:46:04.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): my global_id is 4297 2024-09-17T15:46:04.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc04003520 con 0x7fcc200a5520 2024-09-17T15:46:04.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1205888160 0 0) 0x7fcc20134b00 con 0x7fcc20135b50 2024-09-17T15:46:04.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): my global_id is 4245 2024-09-17T15:46:04.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc2009ccd0 con 0x7fcc20135b50 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3461165919 0 0) 0x7fcc200a8ef0 con 0x7fcc20135f70 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): my global_id is 4273 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc20134b00 con 0x7fcc20135f70 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 638094779 0 0) 0x7fcc04003520 con 0x7fcc200a5520 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc200a8ef0 con 0x7fcc200a5520 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcc1c0047e0 con 0x7fcc200a5520 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): dump: 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.407 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.408 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.408 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.408 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.408 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.408 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.408 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.408 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.408 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.408 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.408 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.408 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.408 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.408 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 313580755 0 0) 0x7fcc2009ccd0 con 0x7fcc20135b50 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc04003520 con 0x7fcc20135b50 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcc10001a10 con 0x7fcc20135b50 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): dump: 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.409 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 722556729 0 0) 0x7fcc20134b00 con 0x7fcc20135f70 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc2009ccd0 con 0x7fcc20135f70 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcc18001bf0 con 0x7fcc20135f70 2024-09-17T15:46:04.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient(hunting): dump: 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.411 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3936859886 0 0) 0x7fcc200a8ef0 con 0x7fcc200a5520 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 >> v1:172.21.15.100:6792/0 conn(0x7fcc20135f70 legacy=0x7fcc20132b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 >> v1:172.21.15.64:6792/0 conn(0x7fcc20135b50 legacy=0x7fcc20132450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 monclient: found mon.c 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2009f2f0 con 0x7fcc200a5520 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:04.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:04.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:04.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc27edd640 5 monclient: authenticate success, global_id 4297 2024-09-17T15:46:04.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc27edd640 10 monclient: _renew_subs 2024-09-17T15:46:04.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc27edd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:04.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc27edd640 1 -- 172.21.15.64:0/1945820389 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc200a0530 con 0x7fcc200a5520 2024-09-17T15:46:04.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc27edd640 10 monclient: _renew_subs 2024-09-17T15:46:04.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc27edd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:04.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc27edd640 1 -- 172.21.15.64:0/1945820389 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc201d7160 con 0x7fcc200a5520 2024-09-17T15:46:04.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc1c004c30 con 0x7fcc200a5520 2024-09-17T15:46:04.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:04.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcc1c0051e0 con 0x7fcc200a5520 2024-09-17T15:46:04.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:04.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc17fff640 4 set_mon_vals no callback set 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.391+0000 7fcc167fc640 10 monclient: dump: 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.414 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.415 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.415 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.415 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.415 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.392+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fcc1c011ac0 con 0x7fcc200a5520 2024-09-17T15:46:04.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.392+0000 7fcc27edd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:04.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.392+0000 7fcc27edd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:04.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.392+0000 7fcc27edd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:04.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.392+0000 7fcc27edd640 1 -- 172.21.15.64:0/1945820389 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc08005080 con 0x7fcc200a5520 2024-09-17T15:46:04.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.392+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7fcc1c04e9a0 con 0x7fcc200a5520 2024-09-17T15:46:04.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.393+0000 7fcc25478640 10 monclient: get_auth_request con 0x7fcc04040cc0 auth_method 0 2024-09-17T15:46:04.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.397+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fcc201346f0 con 0x7fcc200a5520 2024-09-17T15:46:04.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.397+0000 7fcc167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:04.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.397+0000 7fcc167fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:04.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.632+0000 7fcc27edd640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-17T15:46:04.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.632+0000 7fcc27edd640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-17T15:46:04.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.632+0000 7fcc27edd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:04.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.632+0000 7fcc27edd640 1 -- 172.21.15.64:0/1945820389 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fcc08005370 con 0x7fcc200a5520 2024-09-17T15:46:04.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.634+0000 7fcc167fc640 1 -- 172.21.15.64:0/1945820389 <== mon.4 v1:172.21.15.64:6791/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+98034 (unknown 481731318 0 2570061654) 0x7fcc1c017740 con 0x7fcc200a5520 2024-09-17T15:46:04.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.634+0000 7fcc167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-17T15:46:04.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.634+0000 7fcc167fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:46:04.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.636+0000 7fcc27edd640 1 -- 172.21.15.64:0/1945820389 >> v1:172.21.15.100:6812/4210963484 conn(0x7fcc04040cc0 legacy=0x7fcc04043180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.636+0000 7fcc27edd640 10 monclient: shutdown 2024-09-17T15:46:04.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.636+0000 7fcc27edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:04.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.636+0000 7fcc27edd640 1 -- 172.21.15.64:0/1945820389 >> v1:172.21.15.64:6791/0 conn(0x7fcc200a5520 legacy=0x7fcc20131d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.636+0000 7fcc27edd640 1 -- 172.21.15.64:0/1945820389 shutdown_connections 2024-09-17T15:46:04.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.636+0000 7fcc27edd640 1 -- 172.21.15.64:0/1945820389 >> 172.21.15.64:0/1945820389 conn(0x7fcc2006db30 msgr2=0x7fcc201334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:04.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.637+0000 7fcc27edd640 1 -- 172.21.15.64:0/1945820389 shutdown_connections 2024-09-17T15:46:04.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.637+0000 7fcc27edd640 1 -- 172.21.15.64:0/1945820389 wait complete. 2024-09-17T15:46:04.648 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":5,"flags":0,"active_gid":4109,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6812","nonce":4210963484}]},"active_addr":"172.21.15.100:6812/4210963484","active_change":"2024-09-17T15:45:47.825550+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":3380808567}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":3421008181}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":3185381748}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":3775430969}]}]} 2024-09-17T15:46:04.651 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-17T15:46:04.651 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-17T15:46:04.651 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T15:46:04.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.750+0000 7f1a29b10640 10 monclient: get_monmap_and_config 2024-09-17T15:46:04.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.750+0000 7f1a29b10640 10 monclient: build_initial_monmap 2024-09-17T15:46:04.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.750+0000 7f1a29b10640 10 monclient: monmap: 2024-09-17T15:46:04.752 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:04.752 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.752 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:04.751294+0000 2024-09-17T15:46:04.752 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:04.751294+0000 2024-09-17T15:46:04.752 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:04.752 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.753 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:04.753 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:04.753 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:04.753 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:04.753 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:04.753 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:04.753 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:04.753 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:04.753 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:04.753 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a29b10640 1 Processor -- start 2024-09-17T15:46:04.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a29b10640 1 -- start start 2024-09-17T15:46:04.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a29b10640 10 monclient: init 2024-09-17T15:46:04.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a29b10640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:04.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a29b10640 10 monclient: _add_conns ranks=[1,3,0,5,8,6,7,4,2] 2024-09-17T15:46:04.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a29b10640 10 monclient(hunting): picked mon.noname-f con 0x7f1a240a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:04.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a29b10640 10 monclient(hunting): picked mon.noname-g con 0x7f1a24135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:04.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a29b10640 10 monclient(hunting): picked mon.noname-a con 0x7f1a24135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:04.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a29b10640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a2409d6f0 con 0x7f1a24135f70 2024-09-17T15:46:04.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a29b10640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a2409d9d0 con 0x7f1a240a5520 2024-09-17T15:46:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a29b10640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a2409dcb0 con 0x7f1a24135b50 2024-09-17T15:46:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a29b10640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a29b10640 10 monclient(hunting): authenticate will time out at 762.426777s 2024-09-17T15:46:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a23fff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f1a24135f70 0x7f1a241415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:38820/0 (socket says 172.21.15.64:38820) 2024-09-17T15:46:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.751+0000 7f1a23fff640 1 -- 172.21.15.64:0/3586826705 learned_addr learned my addr 172.21.15.64:0/3586826705 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 1 -- 172.21.15.64:0/3586826705 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3885201507 0 0) 0x7f1a2409d6f0 con 0x7f1a24135f70 2024-09-17T15:46:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient(hunting): my global_id is 4257 2024-09-17T15:46:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 1 -- 172.21.15.64:0/3586826705 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a0c003520 con 0x7f1a24135f70 2024-09-17T15:46:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 1 -- 172.21.15.64:0/3586826705 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3952682831 0 0) 0x7f1a2409d9d0 con 0x7f1a240a5520 2024-09-17T15:46:04.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient(hunting): my global_id is 4312 2024-09-17T15:46:04.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 1 -- 172.21.15.64:0/3586826705 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a2409d6f0 con 0x7f1a240a5520 2024-09-17T15:46:04.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 1 -- 172.21.15.64:0/3586826705 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 8157961 0 0) 0x7f1a0c003520 con 0x7f1a24135f70 2024-09-17T15:46:04.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 1 -- 172.21.15.64:0/3586826705 >> v1:172.21.15.100:6790/0 conn(0x7f1a24135b50 legacy=0x7f1a2413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 1 -- 172.21.15.64:0/3586826705 >> v1:172.21.15.100:6789/0 conn(0x7f1a240a5520 legacy=0x7f1a240a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:04.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 1 monclient: found mon.noname-a 2024-09-17T15:46:04.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:04.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:46:04.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 1 -- 172.21.15.64:0/3586826705 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a2409ccd0 con 0x7f1a24135f70 2024-09-17T15:46:04.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:04.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:04.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:04.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 1 -- 172.21.15.64:0/3586826705 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a1c001bb0 con 0x7f1a24135f70 2024-09-17T15:46:04.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a29b10640 5 monclient: authenticate success, global_id 4257 2024-09-17T15:46:04.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a29b10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient: dump: 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.765 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a29b10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 1 -- 172.21.15.64:0/3586826705 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a1c001d50 con 0x7f1a24135f70 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 1 -- 172.21.15.64:0/3586826705 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a1c004f60 con 0x7f1a24135f70 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a29b10640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a29b10640 4 set_mon_vals no callback set 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:04.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.752+0000 7f1a227fc640 10 monclient: dump: 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.767 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.768 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.768 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.753+0000 7f1a29b10640 10 monclient: shutdown 2024-09-17T15:46:04.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.753+0000 7f1a29b10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:04.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.753+0000 7f1a29b10640 1 -- 172.21.15.64:0/3586826705 >> v1:172.21.15.64:6789/0 conn(0x7f1a24135f70 legacy=0x7f1a241415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.753+0000 7f1a29b10640 1 -- 172.21.15.64:0/3586826705 shutdown_connections 2024-09-17T15:46:04.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.753+0000 7f1a29b10640 1 -- 172.21.15.64:0/3586826705 >> 172.21.15.64:0/3586826705 conn(0x7f1a2406db30 msgr2=0x7f1a240ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:04.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.753+0000 7f1a29b10640 1 -- 172.21.15.64:0/3586826705 shutdown_connections 2024-09-17T15:46:04.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.753+0000 7f1a29b10640 1 -- 172.21.15.64:0/3586826705 wait complete. 2024-09-17T15:46:04.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.753+0000 7f1a29b10640 10 monclient: build_initial_monmap 2024-09-17T15:46:04.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.753+0000 7f1a29b10640 10 monclient: monmap: 2024-09-17T15:46:04.768 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:04.768 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:04.769 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.753+0000 7f1a29b10640 1 Processor -- start 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a29b10640 1 -- start start 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a29b10640 10 monclient: init 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a29b10640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a29b10640 10 monclient: _add_conns ranks=[1,4,8,0,3,2,5,6,7] 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a29b10640 10 monclient(hunting): picked mon.noname-f con 0x7f1a240a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a29b10640 10 monclient(hunting): picked mon.noname-c con 0x7f1a24135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a29b10640 10 monclient(hunting): picked mon.noname-e con 0x7f1a24135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a29b10640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a2413a8e0 con 0x7f1a240a5520 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a29b10640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a2413abc0 con 0x7f1a24135b50 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a29b10640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a24135270 con 0x7f1a24135f70 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a29b10640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a29b10640 10 monclient(hunting): authenticate will time out at 762.429672s 2024-09-17T15:46:04.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a23fff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f1a24135f70 0x7f1a241d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:39862/0 (socket says 172.21.15.64:39862) 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a22ffd640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f1a24135b50 0x7f1a241375d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:50624/0 (socket says 172.21.15.64:50624) 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a23fff640 1 -- 172.21.15.64:0/1193233120 learned_addr learned my addr 172.21.15.64:0/1193233120 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.754+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1702168567 0 0) 0x7f1a2413abc0 con 0x7f1a24135b50 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): my global_id is 4306 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a10003520 con 0x7f1a24135b50 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3550215873 0 0) 0x7f1a24135270 con 0x7f1a24135f70 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): my global_id is 4247 2024-09-17T15:46:04.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a2413abc0 con 0x7f1a24135f70 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 155160911 0 0) 0x7f1a2413a8e0 con 0x7f1a240a5520 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): my global_id is 4321 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a24135270 con 0x7f1a240a5520 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4035892235 0 0) 0x7f1a10003520 con 0x7f1a24135b50 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a2413a8e0 con 0x7f1a24135b50 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a18001a70 con 0x7f1a24135b50 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): dump: 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.772 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.773 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1906571866 0 0) 0x7f1a2413abc0 con 0x7f1a24135f70 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a10003520 con 0x7f1a24135f70 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2127318105 0 0) 0x7f1a24135270 con 0x7f1a240a5520 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a2413abc0 con 0x7f1a240a5520 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a1c001af0 con 0x7f1a24135f70 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): dump: 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.774 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.775 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.775 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.775 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.775 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.775 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.775 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.775 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.775 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.775 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.775 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a14001a50 con 0x7f1a240a5520 2024-09-17T15:46:04.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:04.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient(hunting): dump: 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.776 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2336919743 0 0) 0x7f1a2413a8e0 con 0x7f1a24135b50 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 >> v1:172.21.15.64:6793/0 conn(0x7f1a24135f70 legacy=0x7f1a241d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 >> v1:172.21.15.100:6789/0 conn(0x7f1a240a5520 legacy=0x7f1a24136ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 monclient: found mon.c 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a240a6760 con 0x7f1a24135b50 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient: _finish_auth 0 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:04.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a29b10640 5 monclient: authenticate success, global_id 4306 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a29b10640 10 monclient: _renew_subs 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a29b10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a29b10640 1 -- 172.21.15.64:0/1193233120 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1a24138860 con 0x7f1a24135b50 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a29b10640 10 monclient: _renew_subs 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a29b10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a29b10640 1 -- 172.21.15.64:0/1193233120 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f1a241393c0 con 0x7f1a24135b50 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a180038c0 con 0x7f1a24135b50 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a18003e70 con 0x7f1a24135b50 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a21ffb640 4 set_mon_vals no callback set 2024-09-17T15:46:04.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.755+0000 7f1a0bfff640 10 monclient: dump: 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:04.779 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:04.780 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:04.780 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:04.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.756+0000 7f1a29b10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:04.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.756+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f1a18004010 con 0x7f1a24135b50 2024-09-17T15:46:04.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.756+0000 7f1a29b10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:04.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.756+0000 7f1a29b10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:04.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.756+0000 7f1a29b10640 1 -- 172.21.15.64:0/1193233120 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a04005080 con 0x7f1a24135b50 2024-09-17T15:46:04.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.757+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f1a1804eac0 con 0x7f1a24135b50 2024-09-17T15:46:04.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.757+0000 7f1a237fe640 10 monclient: get_auth_request con 0x7f1a10040e00 auth_method 0 2024-09-17T15:46:04.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.761+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f1a18017740 con 0x7f1a24135b50 2024-09-17T15:46:04.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.761+0000 7f1a0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:04.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.761+0000 7f1a0bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:04.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.946+0000 7f1a29b10640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:46:04.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.946+0000 7f1a29b10640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:46:04.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.946+0000 7f1a29b10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:04.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.946+0000 7f1a29b10640 1 -- 172.21.15.64:0/1193233120 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a04005370 con 0x7f1a24135b50 2024-09-17T15:46:04.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.947+0000 7f1a0bfff640 1 -- 172.21.15.64:0/1193233120 <== mon.4 v1:172.21.15.64:6791/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10953 (unknown 941590788 0 412723734) 0x7f1a18017040 con 0x7f1a24135b50 2024-09-17T15:46:04.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.947+0000 7f1a0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:46:04.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.947+0000 7f1a0bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:04.949 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:46:04.949 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":19,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","created":"2024-09-17T15:45:43.591977+0000","modified":"2024-09-17T15:46:03.940942+0000","last_up_change":"2024-09-17T15:45:55.895333+0000","last_in_change":"2024-09-17T15:45:45.218655+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T15:45:53.848072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T15:46:00.391921+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1d9b48f9-b24c-453f-a5e9-79d74f56118c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":796819178}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":796819178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":796819178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":796819178}]},"public_addr":"172.21.15.64:6808/796819178","cluster_addr":"172.21.15.64:6809/796819178","heartbeat_back_addr":"172.21.15.64:6811/796819178","heartbeat_front_addr":"172.21.15.64:6810/796819178","state":["exists","up"]},{"osd":1,"uuid":"6b8dcb8b-604c-46d3-a682-5ef3e93155d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6800","nonce":3889666033}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":3889666033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":3889666033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":3889666033}]},"public_addr":"172.21.15.64:6800/3889666033","cluster_addr":"172.21.15.64:6801/3889666033","heartbeat_back_addr":"172.21.15.64:6803/3889666033","heartbeat_front_addr":"172.21.15.64:6802/3889666033","state":["exists","up"]},{"osd":2,"uuid":"34a266c9-43fe-4e7b-ae85-df0115c5f618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":1825798494}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":1825798494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":1825798494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":1825798494}]},"public_addr":"172.21.15.64:6804/1825798494","cluster_addr":"172.21.15.64:6805/1825798494","heartbeat_back_addr":"172.21.15.64:6807/1825798494","heartbeat_front_addr":"172.21.15.64:6806/1825798494","state":["exists","up"]},{"osd":3,"uuid":"c8740bab-d046-474f-b6a4-84333eb1e094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6804","nonce":2791828047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6805","nonce":2791828047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6807","nonce":2791828047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6806","nonce":2791828047}]},"public_addr":"172.21.15.100:6804/2791828047","cluster_addr":"172.21.15.100:6805/2791828047","heartbeat_back_addr":"172.21.15.100:6807/2791828047","heartbeat_front_addr":"172.21.15.100:6806/2791828047","state":["exists","up"]},{"osd":4,"uuid":"08a1700b-63c5-4028-824f-dbd38931ddb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6808","nonce":4242749853}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6809","nonce":4242749853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6811","nonce":4242749853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6810","nonce":4242749853}]},"public_addr":"172.21.15.100:6808/4242749853","cluster_addr":"172.21.15.100:6809/4242749853","heartbeat_back_addr":"172.21.15.100:6811/4242749853","heartbeat_front_addr":"172.21.15.100:6810/4242749853","state":["exists","up"]},{"osd":5,"uuid":"81fa403f-39e9-44fd-8b1f-ef72fd7ba855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6800","nonce":4145568194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6801","nonce":4145568194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6803","nonce":4145568194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6802","nonce":4145568194}]},"public_addr":"172.21.15.100:6800/4145568194","cluster_addr":"172.21.15.100:6801/4145568194","heartbeat_back_addr":"172.21.15.100:6803/4145568194","heartbeat_front_addr":"172.21.15.100:6802/4145568194","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.703397+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.212068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.432455+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.332164+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.533990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.237115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:46:04.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.949+0000 7f1a29b10640 1 -- 172.21.15.64:0/1193233120 >> v1:172.21.15.100:6812/4210963484 conn(0x7f1a10040e00 legacy=0x7f1a100432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.949+0000 7f1a29b10640 10 monclient: shutdown 2024-09-17T15:46:04.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.949+0000 7f1a29b10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:04.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.949+0000 7f1a29b10640 1 -- 172.21.15.64:0/1193233120 >> v1:172.21.15.64:6791/0 conn(0x7f1a24135b50 legacy=0x7f1a241375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:04.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.949+0000 7f1a29b10640 1 -- 172.21.15.64:0/1193233120 shutdown_connections 2024-09-17T15:46:04.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.949+0000 7f1a29b10640 1 -- 172.21.15.64:0/1193233120 >> 172.21.15.64:0/1193233120 conn(0x7f1a2406db30 msgr2=0x7f1a24133dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:04.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.949+0000 7f1a29b10640 1 -- 172.21.15.64:0/1193233120 shutdown_connections 2024-09-17T15:46:04.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:04.949+0000 7f1a29b10640 1 -- 172.21.15.64:0/1193233120 wait complete. 2024-09-17T15:46:04.963 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-17T15:46:04.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T15:46:05.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.062+0000 7f8ab8dec640 10 monclient: get_monmap_and_config 2024-09-17T15:46:05.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.062+0000 7f8ab8dec640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.062+0000 7f8ab8dec640 10 monclient: monmap: 2024-09-17T15:46:05.064 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.064 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:05.063508+0000 2024-09-17T15:46:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:05.063508+0000 2024-09-17T15:46:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.066 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.063+0000 7f8ab8dec640 1 Processor -- start 2024-09-17T15:46:05.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.063+0000 7f8ab8dec640 1 -- start start 2024-09-17T15:46:05.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.063+0000 7f8ab8dec640 10 monclient: init 2024-09-17T15:46:05.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.063+0000 7f8ab8dec640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.063+0000 7f8ab8dec640 10 monclient: _add_conns ranks=[2,8,3,0,5,4,6,7,1] 2024-09-17T15:46:05.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.063+0000 7f8ab8dec640 10 monclient(hunting): picked mon.noname-b con 0x7f8ab40a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:05.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab8dec640 10 monclient(hunting): picked mon.noname-e con 0x7f8ab4135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:05.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab8dec640 10 monclient(hunting): picked mon.noname-g con 0x7f8ab4135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:05.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab8dec640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab409d6f0 con 0x7f8ab40a5520 2024-09-17T15:46:05.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab8dec640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab409d9d0 con 0x7f8ab4135f70 2024-09-17T15:46:05.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab8dec640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab409dcb0 con 0x7f8ab4135b50 2024-09-17T15:46:05.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab8dec640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab8dec640 10 monclient(hunting): authenticate will time out at 762.739148s 2024-09-17T15:46:05.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab259c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f8ab40a5520 0x7f8ab40a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:48194/0 (socket says 172.21.15.64:48194) 2024-09-17T15:46:05.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab259c640 1 -- 172.21.15.64:0/4175896756 learned_addr learned my addr 172.21.15.64:0/4175896756 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8aabfff640 1 --1- 172.21.15.64:0/4175896756 >> v1:172.21.15.64:6793/0 conn(0x7f8ab4135b50 0x7f8ab413deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:39876/0 (socket says 172.21.15.64:39876) 2024-09-17T15:46:05.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 1 -- 172.21.15.64:0/4175896756 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 140942610 0 0) 0x7f8ab409d6f0 con 0x7f8ab40a5520 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient(hunting): my global_id is 4223 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 1 -- 172.21.15.64:0/4175896756 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8aa0003520 con 0x7f8ab40a5520 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 1 -- 172.21.15.64:0/4175896756 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 932886861 0 0) 0x7f8ab409dcb0 con 0x7f8ab4135b50 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient(hunting): my global_id is 4256 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 1 -- 172.21.15.64:0/4175896756 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ab409d6f0 con 0x7f8ab4135b50 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 1 -- 172.21.15.64:0/4175896756 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 578517710 0 0) 0x7f8aa0003520 con 0x7f8ab40a5520 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 1 -- 172.21.15.64:0/4175896756 >> v1:172.21.15.64:6793/0 conn(0x7f8ab4135b50 legacy=0x7f8ab413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 1 -- 172.21.15.64:0/4175896756 >> v1:172.21.15.100:6790/0 conn(0x7f8ab4135f70 legacy=0x7f8ab41415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 1 monclient: found mon.noname-b 2024-09-17T15:46:05.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 1 -- 172.21.15.64:0/4175896756 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab409ccd0 con 0x7f8ab40a5520 2024-09-17T15:46:05.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 1 -- 172.21.15.64:0/4175896756 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8aa4001830 con 0x7f8ab40a5520 2024-09-17T15:46:05.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab8dec640 5 monclient: authenticate success, global_id 4223 2024-09-17T15:46:05.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab8dec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:05.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient: dump: 2024-09-17T15:46:05.076 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.076 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.076 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.076 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.076 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.076 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.076 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.076 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.076 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.076 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.076 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.076 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab8dec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 1 -- 172.21.15.64:0/4175896756 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8aa4001a50 con 0x7f8ab40a5520 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab8dec640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 1 -- 172.21.15.64:0/4175896756 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8aa4001d40 con 0x7f8ab40a5520 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab8dec640 4 set_mon_vals no callback set 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.064+0000 7f8ab1d9b640 10 monclient: dump: 2024-09-17T15:46:05.077 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.078 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.065+0000 7f8ab8dec640 10 monclient: shutdown 2024-09-17T15:46:05.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.065+0000 7f8ab8dec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.065+0000 7f8ab8dec640 1 -- 172.21.15.64:0/4175896756 >> v1:172.21.15.64:6790/0 conn(0x7f8ab40a5520 legacy=0x7f8ab40a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.065+0000 7f8ab8dec640 1 -- 172.21.15.64:0/4175896756 shutdown_connections 2024-09-17T15:46:05.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.065+0000 7f8ab8dec640 1 -- 172.21.15.64:0/4175896756 >> 172.21.15.64:0/4175896756 conn(0x7f8ab406db30 msgr2=0x7f8ab40ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.065+0000 7f8ab8dec640 1 -- 172.21.15.64:0/4175896756 shutdown_connections 2024-09-17T15:46:05.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.065+0000 7f8ab8dec640 1 -- 172.21.15.64:0/4175896756 wait complete. 2024-09-17T15:46:05.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 10 monclient: monmap: 2024-09-17T15:46:05.079 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.079 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:05.079 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:05.079 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 1 Processor -- start 2024-09-17T15:46:05.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 1 -- start start 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 10 monclient: init 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 10 monclient: _add_conns ranks=[3,6,4,7,1,2,0,8,5] 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 10 monclient(hunting): picked mon.noname-g con 0x7f8ab40a5520 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 10 monclient(hunting): picked mon.noname-d con 0x7f8ab4135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab4135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab40a08d0 con 0x7f8ab40a5520 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab40a92b0 con 0x7f8ab4135f70 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab40a45e0 con 0x7f8ab4135b50 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.066+0000 7f8ab8dec640 10 monclient(hunting): authenticate will time out at 762.741929s 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aabfff640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f8ab4135b50 0x7f8ab4132200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:38296/0 (socket says 172.21.15.64:38296) 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aabfff640 1 -- 172.21.15.64:0/3229295915 learned_addr learned my addr 172.21.15.64:0/3229295915 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 852245016 0 0) 0x7f8ab40a45e0 con 0x7f8ab4135b50 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): my global_id is 4254 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a94003520 con 0x7f8ab4135b50 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3309027067 0 0) 0x7f8ab40a92b0 con 0x7f8ab4135f70 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): my global_id is 4315 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ab40a45e0 con 0x7f8ab4135f70 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2554517542 0 0) 0x7f8ab40a08d0 con 0x7f8ab40a5520 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): my global_id is 4287 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ab40a92b0 con 0x7f8ab40a5520 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1473683372 0 0) 0x7f8a94003520 con 0x7f8ab4135b50 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ab40a08d0 con 0x7f8ab4135b50 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8a9c001bb0 con 0x7f8ab4135b50 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): dump: 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.083 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.084 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.084 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.084 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.084 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.084 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.084 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.084 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.084 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.084 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.084 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.084 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:05.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 882458085 0 0) 0x7f8ab40a45e0 con 0x7f8ab4135f70 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8a94003520 con 0x7f8ab4135f70 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8aac001cb0 con 0x7f8ab4135f70 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): dump: 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.086 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.086 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.086 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.086 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.086 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.086 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.086 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3811335739 0 0) 0x7f8ab40a92b0 con 0x7f8ab40a5520 2024-09-17T15:46:05.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ab40a45e0 con 0x7f8ab40a5520 2024-09-17T15:46:05.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8aa40017f0 con 0x7f8ab40a5520 2024-09-17T15:46:05.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:46:05.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient(hunting): dump: 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.087 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.088 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.088 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3763936728 0 0) 0x7f8ab40a08d0 con 0x7f8ab4135b50 2024-09-17T15:46:05.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 >> v1:172.21.15.64:6791/0 conn(0x7f8ab4135f70 legacy=0x7f8ab41d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 >> v1:172.21.15.100:6790/0 conn(0x7f8ab40a5520 legacy=0x7f8ab4131af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 monclient: found mon.d 2024-09-17T15:46:05.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:05.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab4132910 con 0x7f8ab4135b50 2024-09-17T15:46:05.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8ab8dec640 5 monclient: authenticate success, global_id 4254 2024-09-17T15:46:05.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8ab8dec640 10 monclient: _renew_subs 2024-09-17T15:46:05.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8ab8dec640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:05.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8ab8dec640 1 -- 172.21.15.64:0/3229295915 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8ab40a0030 con 0x7f8ab4135b50 2024-09-17T15:46:05.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8ab8dec640 10 monclient: _renew_subs 2024-09-17T15:46:05.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8ab8dec640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:05.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8ab8dec640 1 -- 172.21.15.64:0/3229295915 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f8ab40a4f80 con 0x7f8ab4135b50 2024-09-17T15:46:05.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.067+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8a9c004a90 con 0x7f8ab4135b50 2024-09-17T15:46:05.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.068+0000 7f8aaaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.068+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8a9c004f80 con 0x7f8ab4135b50 2024-09-17T15:46:05.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.068+0000 7f8aaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.068+0000 7f8ab159a640 4 set_mon_vals no callback set 2024-09-17T15:46:05.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.068+0000 7f8aaaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.068+0000 7f8aaaffd640 10 monclient: dump: 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.090 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.091 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.091 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.068+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f8a9c011860 con 0x7f8ab4135b50 2024-09-17T15:46:05.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.068+0000 7f8ab8dec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.068+0000 7f8ab8dec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.068+0000 7f8ab8dec640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:05.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.068+0000 7f8ab8dec640 1 -- 172.21.15.64:0/3229295915 --> v1:172.21.15.64:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a98005080 con 0x7f8ab4135b50 2024-09-17T15:46:05.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.069+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f8a9c04ebb0 con 0x7f8ab4135b50 2024-09-17T15:46:05.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.069+0000 7f8ab259c640 10 monclient: get_auth_request con 0x7f8a94040c40 auth_method 0 2024-09-17T15:46:05.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.073+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.6 v1:172.21.15.64:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f8a9c0177d0 con 0x7f8ab4135b50 2024-09-17T15:46:05.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.073+0000 7f8aaaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.073+0000 7f8aaaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:05.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.258+0000 7f8ab8dec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:46:05.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.258+0000 7f8ab8dec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:46:05.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.258+0000 7f8ab8dec640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:05.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.258+0000 7f8ab8dec640 1 -- 172.21.15.64:0/3229295915 --> v1:172.21.15.64:6792/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a98005370 con 0x7f8ab4135b50 2024-09-17T15:46:05.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.259+0000 7f8aaaffd640 1 -- 172.21.15.64:0/3229295915 <== mon.6 v1:172.21.15.64:6792/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10953 (unknown 941590788 0 412723734) 0x7f8a9c0170d0 con 0x7f8ab4135b50 2024-09-17T15:46:05.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.259+0000 7f8aaaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:46:05.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.259+0000 7f8aaaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:46:05.261 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":19,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","created":"2024-09-17T15:45:43.591977+0000","modified":"2024-09-17T15:46:03.940942+0000","last_up_change":"2024-09-17T15:45:55.895333+0000","last_in_change":"2024-09-17T15:45:45.218655+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T15:45:53.848072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T15:46:00.391921+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1d9b48f9-b24c-453f-a5e9-79d74f56118c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":796819178}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":796819178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":796819178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":796819178}]},"public_addr":"172.21.15.64:6808/796819178","cluster_addr":"172.21.15.64:6809/796819178","heartbeat_back_addr":"172.21.15.64:6811/796819178","heartbeat_front_addr":"172.21.15.64:6810/796819178","state":["exists","up"]},{"osd":1,"uuid":"6b8dcb8b-604c-46d3-a682-5ef3e93155d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6800","nonce":3889666033}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":3889666033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":3889666033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":3889666033}]},"public_addr":"172.21.15.64:6800/3889666033","cluster_addr":"172.21.15.64:6801/3889666033","heartbeat_back_addr":"172.21.15.64:6803/3889666033","heartbeat_front_addr":"172.21.15.64:6802/3889666033","state":["exists","up"]},{"osd":2,"uuid":"34a266c9-43fe-4e7b-ae85-df0115c5f618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":1825798494}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":1825798494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":1825798494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":1825798494}]},"public_addr":"172.21.15.64:6804/1825798494","cluster_addr":"172.21.15.64:6805/1825798494","heartbeat_back_addr":"172.21.15.64:6807/1825798494","heartbeat_front_addr":"172.21.15.64:6806/1825798494","state":["exists","up"]},{"osd":3,"uuid":"c8740bab-d046-474f-b6a4-84333eb1e094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6804","nonce":2791828047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6805","nonce":2791828047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6807","nonce":2791828047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6806","nonce":2791828047}]},"public_addr":"172.21.15.100:6804/2791828047","cluster_addr":"172.21.15.100:6805/2791828047","heartbeat_back_addr":"172.21.15.100:6807/2791828047","heartbeat_front_addr":"172.21.15.100:6806/2791828047","state":["exists","up"]},{"osd":4,"uuid":"08a1700b-63c5-4028-824f-dbd38931ddb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6808","nonce":4242749853}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6809","nonce":4242749853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6811","nonce":4242749853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6810","nonce":4242749853}]},"public_addr":"172.21.15.100:6808/4242749853","cluster_addr":"172.21.15.100:6809/4242749853","heartbeat_back_addr":"172.21.15.100:6811/4242749853","heartbeat_front_addr":"172.21.15.100:6810/4242749853","state":["exists","up"]},{"osd":5,"uuid":"81fa403f-39e9-44fd-8b1f-ef72fd7ba855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6800","nonce":4145568194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6801","nonce":4145568194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6803","nonce":4145568194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6802","nonce":4145568194}]},"public_addr":"172.21.15.100:6800/4145568194","cluster_addr":"172.21.15.100:6801/4145568194","heartbeat_back_addr":"172.21.15.100:6803/4145568194","heartbeat_front_addr":"172.21.15.100:6802/4145568194","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.703397+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.212068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.432455+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.332164+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.533990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.237115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:46:05.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.261+0000 7f8ab8dec640 1 -- 172.21.15.64:0/3229295915 >> v1:172.21.15.100:6812/4210963484 conn(0x7f8a94040c40 legacy=0x7f8a94043100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.261+0000 7f8ab8dec640 10 monclient: shutdown 2024-09-17T15:46:05.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.261+0000 7f8ab8dec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.261+0000 7f8ab8dec640 1 -- 172.21.15.64:0/3229295915 >> v1:172.21.15.64:6792/0 conn(0x7f8ab4135b50 legacy=0x7f8ab4132200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.261+0000 7f8ab8dec640 1 -- 172.21.15.64:0/3229295915 shutdown_connections 2024-09-17T15:46:05.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.261+0000 7f8ab8dec640 1 -- 172.21.15.64:0/3229295915 >> 172.21.15.64:0/3229295915 conn(0x7f8ab406db30 msgr2=0x7f8ab41334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.262+0000 7f8ab8dec640 1 -- 172.21.15.64:0/3229295915 shutdown_connections 2024-09-17T15:46:05.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.262+0000 7f8ab8dec640 1 -- 172.21.15.64:0/3229295915 wait complete. 2024-09-17T15:46:05.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-17T15:46:05.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-17T15:46:05.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-17T15:46:05.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-17T15:46:05.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-17T15:46:05.276 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-17T15:46:05.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.390+0000 7f0e15803640 10 monclient: get_monmap_and_config 2024-09-17T15:46:05.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.390+0000 7f0e15803640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.390+0000 7f0e15803640 10 monclient: monmap: 2024-09-17T15:46:05.392 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.392 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.392 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:05.391296+0000 2024-09-17T15:46:05.392 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:05.391296+0000 2024-09-17T15:46:05.392 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.393 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.393 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.393 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.393 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.393 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.393 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.393 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.393 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.393 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.393 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.393 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.391+0000 7f0e15803640 1 Processor -- start 2024-09-17T15:46:05.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.391+0000 7f0e15803640 1 -- start start 2024-09-17T15:46:05.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.391+0000 7f0e15803640 10 monclient: init 2024-09-17T15:46:05.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.391+0000 7f0e15803640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.391+0000 7f0e15803640 10 monclient: _add_conns ranks=[5,3,2,1,4,7,0,8,6] 2024-09-17T15:46:05.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.391+0000 7f0e15803640 10 monclient(hunting): picked mon.noname-h con 0x7f0e100a54d0 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:05.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.391+0000 7f0e15803640 10 monclient(hunting): picked mon.noname-g con 0x7f0e10135f40 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:05.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.391+0000 7f0e15803640 10 monclient(hunting): picked mon.noname-b con 0x7f0e1013f470 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:05.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.391+0000 7f0e15803640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e1009d6a0 con 0x7f0e1013f470 2024-09-17T15:46:05.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.391+0000 7f0e15803640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e1009d980 con 0x7f0e10135f40 2024-09-17T15:46:05.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.391+0000 7f0e15803640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e1009dc60 con 0x7f0e100a54d0 2024-09-17T15:46:05.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.391+0000 7f0e15803640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.391+0000 7f0e15803640 10 monclient(hunting): authenticate will time out at 763.066978s 2024-09-17T15:46:05.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0f7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f0e1013f470 0x7f0e10141860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46722/0 (socket says 172.21.15.64:46722) 2024-09-17T15:46:05.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0f7fe640 1 -- 172.21.15.64:0/1784642133 learned_addr learned my addr 172.21.15.64:0/1784642133 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 1 -- 172.21.15.64:0/1784642133 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2618291965 0 0) 0x7f0e1009d980 con 0x7f0e10135f40 2024-09-17T15:46:05.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 10 monclient(hunting): my global_id is 4296 2024-09-17T15:46:05.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 1 -- 172.21.15.64:0/1784642133 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0df8003520 con 0x7f0e10135f40 2024-09-17T15:46:05.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 1 -- 172.21.15.64:0/1784642133 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 963401254 0 0) 0x7f0e1009dc60 con 0x7f0e100a54d0 2024-09-17T15:46:05.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 10 monclient(hunting): my global_id is 4244 2024-09-17T15:46:05.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 1 -- 172.21.15.64:0/1784642133 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e1009d980 con 0x7f0e100a54d0 2024-09-17T15:46:05.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 1 -- 172.21.15.64:0/1784642133 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 986975196 0 0) 0x7f0e1009d6a0 con 0x7f0e1013f470 2024-09-17T15:46:05.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 10 monclient(hunting): my global_id is 4232 2024-09-17T15:46:05.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.392+0000 7f0e0dffb640 1 -- 172.21.15.64:0/1784642133 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e1009dc60 con 0x7f0e1013f470 2024-09-17T15:46:05.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 1 -- 172.21.15.64:0/1784642133 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 422968886 0 0) 0x7f0df8003520 con 0x7f0e10135f40 2024-09-17T15:46:05.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 1 -- 172.21.15.64:0/1784642133 >> v1:172.21.15.100:6791/0 conn(0x7f0e100a54d0 legacy=0x7f0e100a58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 1 -- 172.21.15.64:0/1784642133 >> v1:172.21.15.64:6790/0 conn(0x7f0e1013f470 legacy=0x7f0e10141860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 1 monclient: found mon.noname-g 2024-09-17T15:46:05.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.100:6790/0 2024-09-17T15:46:05.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 1 -- 172.21.15.64:0/1784642133 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e1009df40 con 0x7f0e10135f40 2024-09-17T15:46:05.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 1 -- 172.21.15.64:0/1784642133 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0e04001760 con 0x7f0e10135f40 2024-09-17T15:46:05.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:46:05.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 10 monclient: dump: 2024-09-17T15:46:05.404 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.404 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.404 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.404 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.404 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.405 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.405 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.405 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.405 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.405 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.405 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.405 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.405 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.405 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.405 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.405 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e15803640 5 monclient: authenticate success, global_id 4296 2024-09-17T15:46:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e15803640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 1 -- 172.21.15.64:0/1784642133 <== mon.3 v1:172.21.15.100:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0e04001900 con 0x7f0e10135f40 2024-09-17T15:46:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 1 -- 172.21.15.64:0/1784642133 <== mon.3 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0e04003610 con 0x7f0e10135f40 2024-09-17T15:46:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:46:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e0dffb640 10 monclient: dump: 2024-09-17T15:46:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e15803640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.393+0000 7f0e15803640 4 set_mon_vals no callback set 2024-09-17T15:46:05.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.394+0000 7f0e15803640 10 monclient: shutdown 2024-09-17T15:46:05.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.394+0000 7f0e15803640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.394+0000 7f0e15803640 1 -- 172.21.15.64:0/1784642133 >> v1:172.21.15.100:6790/0 conn(0x7f0e10135f40 legacy=0x7f0e1013dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.394+0000 7f0e15803640 1 -- 172.21.15.64:0/1784642133 shutdown_connections 2024-09-17T15:46:05.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.394+0000 7f0e15803640 1 -- 172.21.15.64:0/1784642133 >> 172.21.15.64:0/1784642133 conn(0x7f0e1006d9a0 msgr2=0x7f0e100ab8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.394+0000 7f0e15803640 1 -- 172.21.15.64:0/1784642133 shutdown_connections 2024-09-17T15:46:05.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.394+0000 7f0e15803640 1 -- 172.21.15.64:0/1784642133 wait complete. 2024-09-17T15:46:05.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.394+0000 7fd2b8952640 10 monclient: get_monmap_and_config 2024-09-17T15:46:05.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.394+0000 7fd2b8952640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.394+0000 7fd2b8952640 10 monclient: monmap: 2024-09-17T15:46:05.409 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.409 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.409 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:05.395792+0000 2024-09-17T15:46:05.409 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:05.395792+0000 2024-09-17T15:46:05.409 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.409 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.409 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.409 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.409 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.409 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.410 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.410 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.410 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.410 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.410 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.410 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.395+0000 7f0e15803640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.395+0000 7f0e15803640 10 monclient: monmap: 2024-09-17T15:46:05.410 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.410 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:05.411 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:05.411 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:05.411 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.411 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:05.411 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.411 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.411 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.411 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.411 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.411 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.411 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.411 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.411 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.412 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.395+0000 7fd2b8952640 1 Processor -- start 2024-09-17T15:46:05.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.395+0000 7fd2b8952640 1 -- start start 2024-09-17T15:46:05.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.395+0000 7fd2b8952640 10 monclient: init 2024-09-17T15:46:05.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.395+0000 7f0e15803640 1 Processor -- start 2024-09-17T15:46:05.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7f0e15803640 1 -- start start 2024-09-17T15:46:05.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7f0e15803640 10 monclient: init 2024-09-17T15:46:05.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7fd2b8952640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7fd2b8952640 10 monclient: _add_conns ranks=[2,7,3,5,6,4,1,0,8] 2024-09-17T15:46:05.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7fd2b8952640 10 monclient(hunting): picked mon.noname-b con 0x7fd2b40a3fe0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:05.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7fd2b8952640 10 monclient(hunting): picked mon.noname-i con 0x7fd2b40a43c0 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:05.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7f0e15803640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7f0e15803640 10 monclient: _add_conns ranks=[2,4,5,8,0,6,7,1,3] 2024-09-17T15:46:05.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7fd2b8952640 10 monclient(hunting): picked mon.noname-g con 0x7fd2b413bfb0 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:05.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7fd2b259c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fd2b40a3fe0 0x7fd2b40a2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46724/0 (socket says 172.21.15.64:46724) 2024-09-17T15:46:05.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7fd2b259c640 1 -- 172.21.15.64:0/908134695 learned_addr learned my addr 172.21.15.64:0/908134695 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7f0e15803640 10 monclient(hunting): picked mon.noname-b con 0x7f0e100a54d0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:05.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7f0e0effd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f0e100a54d0 0x7f0e100a1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46736/0 (socket says 172.21.15.64:46736) 2024-09-17T15:46:05.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7f0e15803640 10 monclient(hunting): picked mon.noname-c con 0x7f0e10135f40 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:05.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.396+0000 7fd2b8952640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2b4069df0 con 0x7fd2b40a3fe0 2024-09-17T15:46:05.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0e15803640 10 monclient(hunting): picked mon.noname-h con 0x7f0e1013f470 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:05.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0e15803640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e1013e490 con 0x7f0e100a54d0 2024-09-17T15:46:05.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0e15803640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e100a8d70 con 0x7f0e10135f40 2024-09-17T15:46:05.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0e0effd640 1 -- 172.21.15.64:0/4085022005 learned_addr learned my addr 172.21.15.64:0/4085022005 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0e15803640 1 -- 172.21.15.64:0/4085022005 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e10142c10 con 0x7f0e1013f470 2024-09-17T15:46:05.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0e15803640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0e15803640 10 monclient(hunting): authenticate will time out at 763.072196s 2024-09-17T15:46:05.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b8952640 1 -- 172.21.15.64:0/908134695 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2b40a1080 con 0x7fd2b413bfb0 2024-09-17T15:46:05.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b8952640 1 -- 172.21.15.64:0/908134695 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2b40a1360 con 0x7fd2b40a43c0 2024-09-17T15:46:05.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 525333624 0 0) 0x7f0e10142c10 con 0x7f0e1013f470 2024-09-17T15:46:05.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b8952640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b8952640 10 monclient(hunting): authenticate will time out at 763.072340s 2024-09-17T15:46:05.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 1 -- 172.21.15.64:0/908134695 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2215285120 0 0) 0x7fd2b40a1080 con 0x7fd2b413bfb0 2024-09-17T15:46:05.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 10 monclient(hunting): my global_id is 4305 2024-09-17T15:46:05.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 1 -- 172.21.15.64:0/908134695 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd29c005730 con 0x7fd2b413bfb0 2024-09-17T15:46:05.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 1 -- 172.21.15.64:0/908134695 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2616719176 0 0) 0x7fd2b40a1360 con 0x7fd2b40a43c0 2024-09-17T15:46:05.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 10 monclient(hunting): my global_id is 4282 2024-09-17T15:46:05.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 1 -- 172.21.15.64:0/908134695 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2b40a1080 con 0x7fd2b40a43c0 2024-09-17T15:46:05.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 10 monclient(hunting): my global_id is 4253 2024-09-17T15:46:05.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dfc003550 con 0x7f0e1013f470 2024-09-17T15:46:05.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 1 -- 172.21.15.64:0/908134695 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3099619765 0 0) 0x7fd2b4069df0 con 0x7fd2b40a3fe0 2024-09-17T15:46:05.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 10 monclient(hunting): my global_id is 4241 2024-09-17T15:46:05.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7fd2b159a640 1 -- 172.21.15.64:0/908134695 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2b40a1360 con 0x7fd2b40a3fe0 2024-09-17T15:46:05.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3126854564 0 0) 0x7f0e1009df40 con 0x7f0e10135f40 2024-09-17T15:46:05.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 10 monclient(hunting): my global_id is 4324 2024-09-17T15:46:05.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e10142c10 con 0x7f0e10135f40 2024-09-17T15:46:05.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1791375166 0 0) 0x7f0e1013e490 con 0x7f0e100a54d0 2024-09-17T15:46:05.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 1 -- 172.21.15.64:0/908134695 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4111348394 0 0) 0x7fd29c005730 con 0x7fd2b413bfb0 2024-09-17T15:46:05.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 1 -- 172.21.15.64:0/908134695 >> v1:172.21.15.100:6792/0 conn(0x7fd2b40a43c0 legacy=0x7fd2b40a47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 1 -- 172.21.15.64:0/908134695 >> v1:172.21.15.64:6790/0 conn(0x7fd2b40a3fe0 legacy=0x7fd2b40a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 1 monclient: found mon.noname-g 2024-09-17T15:46:05.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.100:6790/0 2024-09-17T15:46:05.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 1 -- 172.21.15.64:0/908134695 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b40a1640 con 0x7fd2b413bfb0 2024-09-17T15:46:05.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 10 monclient(hunting): my global_id is 4250 2024-09-17T15:46:05.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.397+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e1009df40 con 0x7f0e100a54d0 2024-09-17T15:46:05.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 1 -- 172.21.15.64:0/908134695 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd2ac001bb0 con 0x7fd2b413bfb0 2024-09-17T15:46:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:46:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 10 monclient: dump: 2024-09-17T15:46:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.426 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.426 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.426 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.426 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.426 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.426 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.426 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 954001283 0 0) 0x7f0e1009df40 con 0x7f0e100a54d0 2024-09-17T15:46:05.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0dfc004db0 con 0x7f0e100a54d0 2024-09-17T15:46:05.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0e00001470 con 0x7f0e100a54d0 2024-09-17T15:46:05.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 10 monclient(hunting): dump: 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.427 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.428 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.428 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.428 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:05.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b8952640 5 monclient: authenticate success, global_id 4305 2024-09-17T15:46:05.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 788988465 0 0) 0x7f0e10142c10 con 0x7f0e10135f40 2024-09-17T15:46:05.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0dfc003cf0 con 0x7f0e10135f40 2024-09-17T15:46:05.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0e04001620 con 0x7f0e10135f40 2024-09-17T15:46:05.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:05.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 10 monclient(hunting): dump: 2024-09-17T15:46:05.429 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.429 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.429 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.429 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.429 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.429 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.429 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.429 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.429 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.430 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.430 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.430 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.430 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.430 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.430 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.430 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b8952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3501565058 0 0) 0x7f0dfc003cf0 con 0x7f0e10135f40 2024-09-17T15:46:05.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 >> v1:172.21.15.100:6791/0 conn(0x7f0e1013f470 legacy=0x7f0e101d5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 >> v1:172.21.15.64:6790/0 conn(0x7f0e100a54d0 legacy=0x7f0e100a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 1 monclient: found mon.c 2024-09-17T15:46:05.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 1 -- 172.21.15.64:0/908134695 <== mon.3 v1:172.21.15.100:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd2ac001d50 con 0x7fd2b413bfb0 2024-09-17T15:46:05.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 1 -- 172.21.15.64:0/908134695 <== mon.3 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd2ac003cb0 con 0x7fd2b413bfb0 2024-09-17T15:46:05.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:46:05.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b159a640 10 monclient: dump: 2024-09-17T15:46:05.432 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.432 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.432 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.432 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.432 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.433 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.433 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.433 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.433 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.433 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.433 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.433 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.433 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.433 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.433 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.433 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:05.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e100a4df0 con 0x7f0e10135f40 2024-09-17T15:46:05.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0df7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b8952640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:05.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7fd2b8952640 4 set_mon_vals no callback set 2024-09-17T15:46:05.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0e15803640 5 monclient: authenticate success, global_id 4324 2024-09-17T15:46:05.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0e15803640 10 monclient: _renew_subs 2024-09-17T15:46:05.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0e15803640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:05.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.398+0000 7f0e15803640 1 -- 172.21.15.64:0/4085022005 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0e100a5130 con 0x7f0e10135f40 2024-09-17T15:46:05.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.399+0000 7f0e15803640 10 monclient: _renew_subs 2024-09-17T15:46:05.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.399+0000 7f0e15803640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:05.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.399+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0e040035f0 con 0x7f0e10135f40 2024-09-17T15:46:05.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.399+0000 7f0e15803640 1 -- 172.21.15.64:0/4085022005 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f0e100a33f0 con 0x7f0e10135f40 2024-09-17T15:46:05.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.399+0000 7f0df7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.399+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0e04003ec0 con 0x7f0e10135f40 2024-09-17T15:46:05.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.399+0000 7f0df7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.399+0000 7f0df7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:05.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.399+0000 7f0df7fff640 10 monclient: dump: 2024-09-17T15:46:05.437 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.437 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.437 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.437 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.437 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.437 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.437 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.437 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.437 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.437 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.437 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.438 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.438 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.438 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.438 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.438 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.399+0000 7f0e0d7fa640 4 set_mon_vals no callback set 2024-09-17T15:46:05.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.399+0000 7fd2b8952640 10 monclient: shutdown 2024-09-17T15:46:05.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.399+0000 7fd2b8952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.399+0000 7fd2b8952640 1 -- 172.21.15.64:0/908134695 >> v1:172.21.15.100:6790/0 conn(0x7fd2b413bfb0 legacy=0x7fd2b413e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.400+0000 7fd2b8952640 1 -- 172.21.15.64:0/908134695 shutdown_connections 2024-09-17T15:46:05.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.400+0000 7fd2b8952640 1 -- 172.21.15.64:0/908134695 >> 172.21.15.64:0/908134695 conn(0x7fd2b40a50d0 msgr2=0x7fd2b40a54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.400+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f0e040107a0 con 0x7f0e10135f40 2024-09-17T15:46:05.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.400+0000 7fd2b8952640 1 -- 172.21.15.64:0/908134695 shutdown_connections 2024-09-17T15:46:05.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.400+0000 7fd2b8952640 1 -- 172.21.15.64:0/908134695 wait complete. 2024-09-17T15:46:05.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.400+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f0e0404d730 con 0x7f0e10135f40 2024-09-17T15:46:05.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.401+0000 7f0e15803640 1 -- 172.21.15.64:0/4085022005 --> v1:172.21.15.64:6808/796819178 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e101366a0 con 0x7f0e100a7c80 2024-09-17T15:46:05.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.401+0000 7f0e0effd640 10 monclient: get_auth_request con 0x7f0dfc041170 auth_method 0 2024-09-17T15:46:05.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.401+0000 7f0e0f7fe640 10 monclient: get_auth_request con 0x7f0e100a7c80 auth_method 0 2024-09-17T15:46:05.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.401+0000 7fd2b8952640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.401+0000 7fd2b8952640 10 monclient: monmap: 2024-09-17T15:46:05.441 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.441 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:05.441 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:05.441 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:05.441 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.402+0000 7fd2b8952640 1 Processor -- start 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.402+0000 7fd2b8952640 1 -- start start 2024-09-17T15:46:05.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.402+0000 7fd2b8952640 10 monclient: init 2024-09-17T15:46:05.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.402+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== osd.0 v1:172.21.15.64:6808/796819178 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0e080018d0 con 0x7f0e100a7c80 2024-09-17T15:46:05.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.402+0000 7fd2b8952640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.402+0000 7fd2b8952640 10 monclient: _add_conns ranks=[3,8,2,5,7,0,1,6,4] 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.402+0000 7fd2b8952640 10 monclient(hunting): picked mon.noname-g con 0x7fd2b40a3fe0 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.402+0000 7fd2b8952640 10 monclient(hunting): picked mon.noname-e con 0x7fd2b40a43c0 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.402+0000 7fd2b8952640 10 monclient(hunting): picked mon.noname-b con 0x7fd2b413bfb0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.402+0000 7fd2b8952640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2b412e9e0 con 0x7fd2b413bfb0 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.402+0000 7fd2b2d9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fd2b413bfb0 0x7fd2b41d2180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46752/0 (socket says 172.21.15.64:46752) 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.402+0000 7fd2b2d9d640 1 -- 172.21.15.64:0/597711923 learned_addr learned my addr 172.21.15.64:0/597711923 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2b8952640 1 -- 172.21.15.64:0/597711923 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2b409ef60 con 0x7fd2b40a3fe0 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3921835664 0 0) 0x7fd2b412e9e0 con 0x7fd2b413bfb0 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2b8952640 1 -- 172.21.15.64:0/597711923 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2b40694b0 con 0x7fd2b40a43c0 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2b8952640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2b8952640 10 monclient(hunting): authenticate will time out at 763.078478s 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2a2ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2a2ffd640 10 monclient(hunting): my global_id is 4259 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd294003520 con 0x7fd2b413bfb0 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2812220183 0 0) 0x7fd294003520 con 0x7fd2b413bfb0 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2b412e9e0 con 0x7fd2b413bfb0 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd2ac001af0 con 0x7fd2b413bfb0 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2a2ffd640 10 monclient(hunting): dump: 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.451 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.452 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.452 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.452 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.452 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.452 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.452 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.452 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.452 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.452 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.452 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.403+0000 7fd2a2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:05.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1380276171 0 0) 0x7fd2b40694b0 con 0x7fd2b40a43c0 2024-09-17T15:46:05.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 10 monclient(hunting): my global_id is 4265 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd294003520 con 0x7fd2b40a43c0 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3705611538 0 0) 0x7fd2b412e9e0 con 0x7fd2b413bfb0 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 >> v1:172.21.15.64:6793/0 conn(0x7fd2b40a43c0 legacy=0x7fd2b4139010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 >> v1:172.21.15.100:6790/0 conn(0x7fd2b40a3fe0 legacy=0x7fd2b4138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 1 monclient: found mon.b 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b40a4de0 con 0x7fd2b413bfb0 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2b8952640 5 monclient: authenticate success, global_id 4259 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2b8952640 10 monclient: _renew_subs 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2b8952640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2b8952640 1 -- 172.21.15.64:0/597711923 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b412e700 con 0x7fd2b413bfb0 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2b8952640 10 monclient: _renew_subs 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2b8952640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.404+0000 7fd2b8952640 1 -- 172.21.15.64:0/597711923 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd2b4139720 con 0x7fd2b413bfb0 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.405+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd2ac001c90 con 0x7fd2b413bfb0 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.405+0000 7fd2a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.405+0000 7fd2b0d99640 4 set_mon_vals no callback set 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.405+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd2ac003f60 con 0x7fd2b413bfb0 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.405+0000 7fd2a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.405+0000 7fd2a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.405+0000 7fd2a2ffd640 10 monclient: dump: 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.455 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.456 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.456 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.405+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fd2ac004180 con 0x7fd2b413bfb0 2024-09-17T15:46:05.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.406+0000 7fd2b8952640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.406+0000 7fd2b8952640 1 -- 172.21.15.64:0/597711923 --> v1:172.21.15.64:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fd298000f80 con 0x7fd2b413bfb0 2024-09-17T15:46:05.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.406+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7fd2ac04ede0 con 0x7fd2b413bfb0 2024-09-17T15:46:05.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.406+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 --> v1:172.21.15.100:6800/4145568194 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd29404ac30 con 0x7fd294046f80 2024-09-17T15:46:05.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.406+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7fd2ac013ac0 con 0x7fd2b413bfb0 2024-09-17T15:46:05.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.406+0000 7fd2a2ffd640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-17T15:46:05.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.406+0000 7fd2b259c640 10 monclient: get_auth_request con 0x7fd2940403f0 auth_method 0 2024-09-17T15:46:05.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.406+0000 7fd2b1d9b640 10 monclient: get_auth_request con 0x7fd294046f80 auth_method 0 2024-09-17T15:46:05.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.407+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 <== osd.5 v1:172.21.15.100:6800/4145568194 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd29404ac30 con 0x7fd294046f80 2024-09-17T15:46:05.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.418+0000 7f0e15803640 1 -- 172.21.15.64:0/4085022005 --> v1:172.21.15.64:6808/796819178 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0e100a37a0 con 0x7f0e100a7c80 2024-09-17T15:46:05.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.421+0000 7f0df7fff640 1 -- 172.21.15.64:0/4085022005 <== osd.0 v1:172.21.15.64:6808/796819178 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3931925601) 0x7f0e100a37a0 con 0x7f0e100a7c80 2024-09-17T15:46:05.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.421+0000 7f0e15803640 1 -- 172.21.15.64:0/4085022005 >> v1:172.21.15.64:6808/796819178 conn(0x7f0e100a7c80 legacy=0x7f0e10063590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.421+0000 7f0e15803640 1 -- 172.21.15.64:0/4085022005 >> v1:172.21.15.100:6812/4210963484 conn(0x7f0dfc041170 legacy=0x7f0dfc043630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.421+0000 7f0e15803640 10 monclient: shutdown 2024-09-17T15:46:05.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.421+0000 7f0e15803640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.421+0000 7f0e15803640 1 -- 172.21.15.64:0/4085022005 >> v1:172.21.15.64:6791/0 conn(0x7f0e10135f40 legacy=0x7f0e101d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.421+0000 7fd2b8952640 1 -- 172.21.15.64:0/597711923 --> v1:172.21.15.100:6800/4145568194 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd298002c70 con 0x7fd294046f80 2024-09-17T15:46:05.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.421+0000 7f0e0f7fe640 1 -- 172.21.15.64:0/4085022005 reap_dead start 2024-09-17T15:46:05.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.421+0000 7f0e15803640 1 -- 172.21.15.64:0/4085022005 shutdown_connections 2024-09-17T15:46:05.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.421+0000 7f0e15803640 1 -- 172.21.15.64:0/4085022005 >> 172.21.15.64:0/4085022005 conn(0x7f0e1006d9a0 msgr2=0x7f0e100a7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.422+0000 7f0e15803640 1 -- 172.21.15.64:0/4085022005 shutdown_connections 2024-09-17T15:46:05.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.422+0000 7f0e15803640 1 -- 172.21.15.64:0/4085022005 wait complete. 2024-09-17T15:46:05.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.422+0000 7fd2a2ffd640 1 -- 172.21.15.64:0/597711923 <== osd.5 v1:172.21.15.100:6800/4145568194 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3580782990) 0x7fd298002c70 con 0x7fd294046f80 2024-09-17T15:46:05.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.422+0000 7fd2b8952640 1 -- 172.21.15.64:0/597711923 >> v1:172.21.15.100:6800/4145568194 conn(0x7fd294046f80 legacy=0x7fd2940493c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.422+0000 7fd2b8952640 1 -- 172.21.15.64:0/597711923 >> v1:172.21.15.100:6812/4210963484 conn(0x7fd2940403f0 legacy=0x7fd2940428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.423+0000 7fd2b8952640 10 monclient: shutdown 2024-09-17T15:46:05.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.423+0000 7fd2b8952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.423+0000 7fd2b8952640 1 -- 172.21.15.64:0/597711923 >> v1:172.21.15.64:6790/0 conn(0x7fd2b413bfb0 legacy=0x7fd2b41d2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.423+0000 7fd2b2d9d640 1 -- 172.21.15.64:0/597711923 reap_dead start 2024-09-17T15:46:05.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.423+0000 7fd2b8952640 1 -- 172.21.15.64:0/597711923 shutdown_connections 2024-09-17T15:46:05.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.423+0000 7fd2b8952640 1 -- 172.21.15.64:0/597711923 >> 172.21.15.64:0/597711923 conn(0x7fd2b40a50d0 msgr2=0x7fd2b413b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.423+0000 7fd2b8952640 1 -- 172.21.15.64:0/597711923 shutdown_connections 2024-09-17T15:46:05.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.423+0000 7fd2b8952640 1 -- 172.21.15.64:0/597711923 wait complete. 2024-09-17T15:46:05.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.432+0000 7f29e34c6640 10 monclient: get_monmap_and_config 2024-09-17T15:46:05.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.432+0000 7f29e34c6640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.432+0000 7f29e34c6640 10 monclient: monmap: 2024-09-17T15:46:05.462 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.462 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.462 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:05.433444+0000 2024-09-17T15:46:05.462 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:05.433444+0000 2024-09-17T15:46:05.462 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.462 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.462 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.463 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.463 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.463 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.463 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.463 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.463 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.463 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.463 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.463 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.433+0000 7f29e34c6640 1 Processor -- start 2024-09-17T15:46:05.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.433+0000 7f29e34c6640 1 -- start start 2024-09-17T15:46:05.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.433+0000 7f29e34c6640 10 monclient: init 2024-09-17T15:46:05.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f2ea9f5d640 10 monclient: get_monmap_and_config 2024-09-17T15:46:05.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f2ea9f5d640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f2ea9f5d640 10 monclient: monmap: 2024-09-17T15:46:05.464 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.464 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.464 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:05.434928+0000 2024-09-17T15:46:05.464 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:05.434928+0000 2024-09-17T15:46:05.464 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.464 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.465 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.465 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.465 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.465 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.465 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.465 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.465 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.465 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.465 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.465 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f29e34c6640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f29e34c6640 10 monclient: _add_conns ranks=[3,8,5,7,1,2,4,0,6] 2024-09-17T15:46:05.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f29e34c6640 10 monclient(hunting): picked mon.noname-g con 0x7f29dc0a54b0 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:05.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f29e34c6640 10 monclient(hunting): picked mon.noname-e con 0x7f29dc135c80 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:05.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f29e34c6640 10 monclient(hunting): picked mon.noname-h con 0x7f29dc1360a0 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:05.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f29e0a61640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f29dc135c80 0x7f29dc13df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38572/0 (socket says 172.21.15.64:38572) 2024-09-17T15:46:05.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f29e34c6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29dc0aacb0 con 0x7f29dc0a54b0 2024-09-17T15:46:05.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f2ea9f5d640 1 Processor -- start 2024-09-17T15:46:05.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f29e0a61640 1 -- 172.21.15.64:0/442078035 learned_addr learned my addr 172.21.15.64:0/442078035 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f29e1262640 1 --1- 172.21.15.64:0/442078035 >> v1:172.21.15.100:6790/0 conn(0x7f29dc0a54b0 0x7f29dc0a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:47720/0 (socket says 172.21.15.64:47720) 2024-09-17T15:46:05.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f29e34c6640 1 -- 172.21.15.64:0/442078035 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29dc09d780 con 0x7f29dc1360a0 2024-09-17T15:46:05.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.434+0000 7f29e34c6640 1 -- 172.21.15.64:0/442078035 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29dc09da60 con 0x7f29dc135c80 2024-09-17T15:46:05.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29e34c6640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29e34c6640 10 monclient(hunting): authenticate will time out at 763.110022s 2024-09-17T15:46:05.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 -- 172.21.15.64:0/442078035 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3891172872 0 0) 0x7f29dc09d780 con 0x7f29dc1360a0 2024-09-17T15:46:05.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient(hunting): my global_id is 4262 2024-09-17T15:46:05.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 -- 172.21.15.64:0/442078035 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29c8003520 con 0x7f29dc1360a0 2024-09-17T15:46:05.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 -- 172.21.15.64:0/442078035 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 877871363 0 0) 0x7f29dc0aacb0 con 0x7f29dc0a54b0 2024-09-17T15:46:05.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient(hunting): my global_id is 4323 2024-09-17T15:46:05.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 -- 172.21.15.64:0/442078035 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29dc09d780 con 0x7f29dc0a54b0 2024-09-17T15:46:05.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 -- 172.21.15.64:0/442078035 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3468336171 0 0) 0x7f29dc09da60 con 0x7f29dc135c80 2024-09-17T15:46:05.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient(hunting): my global_id is 4274 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 -- 172.21.15.64:0/442078035 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29dc0aacb0 con 0x7f29dc135c80 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 -- 172.21.15.64:0/442078035 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3793846338 0 0) 0x7f29c8003520 con 0x7f29dc1360a0 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 -- 172.21.15.64:0/442078035 >> v1:172.21.15.64:6793/0 conn(0x7f29dc135c80 legacy=0x7f29dc13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 -- 172.21.15.64:0/442078035 >> v1:172.21.15.100:6790/0 conn(0x7f29dc0a54b0 legacy=0x7f29dc0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 monclient: found mon.noname-h 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.100:6791/0 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 -- 172.21.15.64:0/442078035 --> v1:172.21.15.100:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29dc09dd40 con 0x7f29dc1360a0 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 -- 172.21.15.64:0/442078035 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29d4001ae0 con 0x7f29dc1360a0 2024-09-17T15:46:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-17T15:46:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient: dump: 2024-09-17T15:46:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.472 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29e34c6640 5 monclient: authenticate success, global_id 4262 2024-09-17T15:46:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29e34c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f2ea9f5d640 1 -- start start 2024-09-17T15:46:05.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f2ea9f5d640 10 monclient: init 2024-09-17T15:46:05.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 -- 172.21.15.64:0/442078035 <== mon.5 v1:172.21.15.100:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f29d4001d00 con 0x7f29dc1360a0 2024-09-17T15:46:05.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 1 -- 172.21.15.64:0/442078035 <== mon.5 v1:172.21.15.100:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29d4003cb0 con 0x7f29dc1360a0 2024-09-17T15:46:05.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:46:05.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29d3fff640 10 monclient: dump: 2024-09-17T15:46:05.473 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.473 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.473 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.473 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29e34c6640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.435+0000 7f29e34c6640 4 set_mon_vals no callback set 2024-09-17T15:46:05.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea9f5d640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea9f5d640 10 monclient: _add_conns ranks=[2,6,0,8,5,7,1,3,4] 2024-09-17T15:46:05.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea9f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ea40a3e00 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:05.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea9f5d640 10 monclient(hunting): picked mon.noname-d con 0x7f2ea40a09b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:05.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea9f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ea413f5b0 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:05.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea9f5d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea40a0590 con 0x7f2ea413f5b0 2024-09-17T15:46:05.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea9f5d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea40a2740 con 0x7f2ea40a3e00 2024-09-17T15:46:05.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea9f5d640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea4069b10 con 0x7f2ea40a09b0 2024-09-17T15:46:05.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea9f5d640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea9f5d640 10 monclient(hunting): authenticate will time out at 763.111324s 2024-09-17T15:46:05.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea2ffd640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f2ea40a09b0 0x7f2ea4068fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:49434/0 (socket says 172.21.15.64:49434) 2024-09-17T15:46:05.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea2ffd640 1 -- 172.21.15.64:0/3316001853 learned_addr learned my addr 172.21.15.64:0/3316001853 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f29e34c6640 10 monclient: shutdown 2024-09-17T15:46:05.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f29e34c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f29e34c6640 1 -- 172.21.15.64:0/442078035 >> v1:172.21.15.100:6791/0 conn(0x7f29dc1360a0 legacy=0x7f29dc141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f29e34c6640 1 -- 172.21.15.64:0/442078035 shutdown_connections 2024-09-17T15:46:05.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f29e34c6640 1 -- 172.21.15.64:0/442078035 >> 172.21.15.64:0/442078035 conn(0x7f29dc06daa0 msgr2=0x7f29dc0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f29e34c6640 1 -- 172.21.15.64:0/442078035 shutdown_connections 2024-09-17T15:46:05.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f29e34c6640 1 -- 172.21.15.64:0/442078035 wait complete. 2024-09-17T15:46:05.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea27fc640 1 -- 172.21.15.64:0/3316001853 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3522757595 0 0) 0x7f2e94001110 con 0x7f2ea40a3e00 2024-09-17T15:46:05.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.436+0000 7f2ea27fc640 10 monclient(hunting): my global_id is 4268 2024-09-17T15:46:05.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 1 -- 172.21.15.64:0/3316001853 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e8c003520 con 0x7f2ea40a3e00 2024-09-17T15:46:05.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 1 -- 172.21.15.64:0/3316001853 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 949398376 0 0) 0x7f2ea40a0590 con 0x7f2ea413f5b0 2024-09-17T15:46:05.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 10 monclient(hunting): my global_id is 4266 2024-09-17T15:46:05.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 1 -- 172.21.15.64:0/3316001853 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e94001110 con 0x7f2ea413f5b0 2024-09-17T15:46:05.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 1 -- 172.21.15.64:0/3316001853 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 869451272 0 0) 0x7f2e94001110 con 0x7f2ea413f5b0 2024-09-17T15:46:05.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 1 -- 172.21.15.64:0/3316001853 >> v1:172.21.15.64:6792/0 conn(0x7f2ea40a09b0 legacy=0x7f2ea4068fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 1 -- 172.21.15.64:0/3316001853 >> v1:172.21.15.64:6790/0 conn(0x7f2ea40a3e00 legacy=0x7f2ea40688b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 1 monclient: found mon.noname-a 2024-09-17T15:46:05.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:46:05.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 1 -- 172.21.15.64:0/3316001853 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea40696d0 con 0x7f2ea413f5b0 2024-09-17T15:46:05.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea9f5d640 5 monclient: authenticate success, global_id 4266 2024-09-17T15:46:05.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea9f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 1 -- 172.21.15.64:0/3316001853 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e9c001ac0 con 0x7f2ea413f5b0 2024-09-17T15:46:05.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f29e34c6640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:05.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea27fc640 10 monclient: dump: 2024-09-17T15:46:05.481 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.481 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.481 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.481 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.481 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.481 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.481 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.482 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.482 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.482 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.482 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.482 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.482 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.482 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.482 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.482 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f29e34c6640 10 monclient: monmap: 2024-09-17T15:46:05.483 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.483 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:05.483 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:05.483 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:05.483 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.483 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:05.483 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.483 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.483 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.483 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.483 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.484 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.484 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.484 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.484 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.484 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.437+0000 7f2ea9f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.485 INFO:teuthology.orchestra.run.smithi064.stdout:60129542152 2024-09-17T15:46:05.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T15:46:05.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f2ea27fc640 1 -- 172.21.15.64:0/3316001853 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2e9c001d20 con 0x7f2ea413f5b0 2024-09-17T15:46:05.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f2ea27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f29e34c6640 1 Processor -- start 2024-09-17T15:46:05.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f29e34c6640 1 -- start start 2024-09-17T15:46:05.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f2ea9f5d640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:05.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f2ea9f5d640 4 set_mon_vals no callback set 2024-09-17T15:46:05.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f2ea27fc640 1 -- 172.21.15.64:0/3316001853 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e9c003990 con 0x7f2ea413f5b0 2024-09-17T15:46:05.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f2ea27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f2ea27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:05.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f29e34c6640 10 monclient: init 2024-09-17T15:46:05.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f2ea27fc640 10 monclient: dump: 2024-09-17T15:46:05.488 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.488 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.488 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.488 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.488 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.488 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f2ea9f5d640 10 monclient: shutdown 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f2ea9f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.438+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/3316001853 >> v1:172.21.15.64:6789/0 conn(0x7f2ea413f5b0 legacy=0x7f2ea41419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/3316001853 shutdown_connections 2024-09-17T15:46:05.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/3316001853 >> 172.21.15.64:0/3316001853 conn(0x7f2ea40a4ef0 msgr2=0x7f2ea40a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/3316001853 shutdown_connections 2024-09-17T15:46:05.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/3316001853 wait complete. 2024-09-17T15:46:05.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f29e34c6640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f29e34c6640 10 monclient: _add_conns ranks=[0,7,1,3,4,2,8,6,5] 2024-09-17T15:46:05.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f29e34c6640 10 monclient(hunting): picked mon.noname-a con 0x7f29dc0a54b0 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:05.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f29e1262640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f29dc0a54b0 0x7f29dc1372d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:58934/0 (socket says 172.21.15.64:58934) 2024-09-17T15:46:05.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f29e34c6640 10 monclient(hunting): picked mon.noname-i con 0x7f29dc135c80 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:05.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f29e34c6640 10 monclient(hunting): picked mon.noname-f con 0x7f29dc1360a0 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:05.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f29e34c6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29dc138ce0 con 0x7f29dc0a54b0 2024-09-17T15:46:05.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f29e34c6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29dc0704c0 con 0x7f29dc1360a0 2024-09-17T15:46:05.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f2ea9f5d640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f2ea9f5d640 10 monclient: monmap: 2024-09-17T15:46:05.492 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.492 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:05.492 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:05.492 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:05.492 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.492 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:05.492 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.492 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.492 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.492 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.493 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.493 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.493 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.493 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.493 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.493 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f29e34c6640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29dc1380f0 con 0x7f29dc135c80 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f29e34c6640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f29e34c6640 10 monclient(hunting): authenticate will time out at 763.114749s 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.439+0000 7f29e1262640 1 -- 172.21.15.64:0/1106961151 learned_addr learned my addr 172.21.15.64:0/1106961151 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3474533428 0 0) 0x7f29dc138ce0 con 0x7f29dc0a54b0 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): my global_id is 4275 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29c0003520 con 0x7f29dc0a54b0 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3268953588 0 0) 0x7f29dc0704c0 con 0x7f29dc1360a0 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): my global_id is 4330 2024-09-17T15:46:05.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29c00048e0 con 0x7f29dc1360a0 2024-09-17T15:46:05.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f2ea9f5d640 1 Processor -- start 2024-09-17T15:46:05.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 681277720 0 0) 0x7f29dc1380f0 con 0x7f29dc135c80 2024-09-17T15:46:05.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): my global_id is 4291 2024-09-17T15:46:05.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29dc0704c0 con 0x7f29dc135c80 2024-09-17T15:46:05.496 INFO:teuthology.orchestra.run.smithi064.stdout:47244640261 2024-09-17T15:46:05.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T15:46:05.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f2ea9f5d640 1 -- start start 2024-09-17T15:46:05.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f2ea9f5d640 10 monclient: init 2024-09-17T15:46:05.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2876003660 0 0) 0x7f29c0003520 con 0x7f29dc0a54b0 2024-09-17T15:46:05.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29dc1380f0 con 0x7f29dc0a54b0 2024-09-17T15:46:05.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29d8001ad0 con 0x7f29dc0a54b0 2024-09-17T15:46:05.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:05.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): dump: 2024-09-17T15:46:05.498 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 555938053 0 0) 0x7f29c00048e0 con 0x7f29dc1360a0 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29c0006e60 con 0x7f29dc1360a0 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1454703714 0 0) 0x7f29dc0704c0 con 0x7f29dc135c80 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29c00048e0 con 0x7f29dc135c80 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29d4001aa0 con 0x7f29dc1360a0 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): dump: 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.500 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29cc0019d0 con 0x7f29dc135c80 2024-09-17T15:46:05.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient(hunting): dump: 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.502 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 737337275 0 0) 0x7f29dc1380f0 con 0x7f29dc0a54b0 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 >> v1:172.21.15.100:6792/0 conn(0x7f29dc135c80 legacy=0x7f29dc1379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 >> v1:172.21.15.100:6789/0 conn(0x7f29dc1360a0 legacy=0x7f29dc1d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 monclient: found mon.a 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29dc13b040 con 0x7f29dc0a54b0 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29e34c6640 5 monclient: authenticate success, global_id 4275 2024-09-17T15:46:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29e34c6640 10 monclient: _renew_subs 2024-09-17T15:46:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29e34c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f29e34c6640 1 -- 172.21.15.64:0/1106961151 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29dc13a7a0 con 0x7f29dc0a54b0 2024-09-17T15:46:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f9d59e60640 10 monclient: get_monmap_and_config 2024-09-17T15:46:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f9d59e60640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.440+0000 7f9d59e60640 10 monclient: monmap: 2024-09-17T15:46:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:05.441675+0000 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:05.441675+0000 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.505 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f29d8004a90 con 0x7f29dc0a54b0 2024-09-17T15:46:05.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f29d1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29d8001ad0 con 0x7f29dc0a54b0 2024-09-17T15:46:05.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f29d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f29d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:05.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f29d1ffb640 10 monclient: dump: 2024-09-17T15:46:05.506 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.506 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.506 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.506 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.506 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.507 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.507 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.507 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.507 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.507 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.507 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.507 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.507 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.507 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.507 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.507 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f2ea9f5d640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f2ea9f5d640 10 monclient: _add_conns ranks=[8,3,2,6,0,1,5,7,4] 2024-09-17T15:46:05.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f2ea9f5d640 10 monclient(hunting): picked mon.noname-e con 0x7f2ea40a09b0 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:05.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f2ea9f5d640 10 monclient(hunting): picked mon.noname-g con 0x7f2ea40a3e00 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:05.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f2ea9f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ea413f5b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:05.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f2ea9f5d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea4073d80 con 0x7f2ea413f5b0 2024-09-17T15:46:05.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f2ea9f5d640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea40a69c0 con 0x7f2ea40a3e00 2024-09-17T15:46:05.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f2ea9f5d640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea413e540 con 0x7f2ea40a09b0 2024-09-17T15:46:05.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f2ea9f5d640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f2ea9f5d640 10 monclient(hunting): authenticate will time out at 763.116311s 2024-09-17T15:46:05.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f29e34c6640 10 monclient: _renew_subs 2024-09-17T15:46:05.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f29e34c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:05.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f29e34c6640 1 -- 172.21.15.64:0/1106961151 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f29dc139ab0 con 0x7f29dc0a54b0 2024-09-17T15:46:05.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f100c27f640 10 monclient: get_monmap_and_config 2024-09-17T15:46:05.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f100c27f640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f100c27f640 10 monclient: monmap: 2024-09-17T15:46:05.509 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.509 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.509 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:05.442346+0000 2024-09-17T15:46:05.509 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:05.442346+0000 2024-09-17T15:46:05.510 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.510 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.510 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.510 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.510 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.510 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.510 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.510 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.510 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.510 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.510 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.510 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f2ea37fe640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f2ea40a09b0 0x7f2ea4137aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38584/0 (socket says 172.21.15.64:38584) 2024-09-17T15:46:05.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f2ea37fe640 1 -- 172.21.15.64:0/600953204 learned_addr learned my addr 172.21.15.64:0/600953204 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f29d37fe640 4 set_mon_vals no callback set 2024-09-17T15:46:05.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.441+0000 7f9d59e60640 1 Processor -- start 2024-09-17T15:46:05.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f9d59e60640 1 -- start start 2024-09-17T15:46:05.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f9d59e60640 10 monclient: init 2024-09-17T15:46:05.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3001528459 0 0) 0x7f2ea413e540 con 0x7f2ea40a09b0 2024-09-17T15:46:05.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): my global_id is 4283 2024-09-17T15:46:05.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e90003520 con 0x7f2ea40a09b0 2024-09-17T15:46:05.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1653293485 0 0) 0x7f2ea4073d80 con 0x7f2ea413f5b0 2024-09-17T15:46:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): my global_id is 4277 2024-09-17T15:46:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ea413e540 con 0x7f2ea413f5b0 2024-09-17T15:46:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2698096850 0 0) 0x7f2ea40a69c0 con 0x7f2ea40a3e00 2024-09-17T15:46:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): my global_id is 4332 2024-09-17T15:46:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ea4073d80 con 0x7f2ea40a3e00 2024-09-17T15:46:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f29e34c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f29e34c6640 1 -- 172.21.15.64:0/1106961151 --> v1:172.21.15.64:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f29dc0a6d00 con 0x7f29dc0a54b0 2024-09-17T15:46:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2728188285 0 0) 0x7f2ea413e540 con 0x7f2ea413f5b0 2024-09-17T15:46:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ea40a69c0 con 0x7f2ea413f5b0 2024-09-17T15:46:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2373767179 0 0) 0x7f2ea4073d80 con 0x7f2ea40a3e00 2024-09-17T15:46:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ea413e540 con 0x7f2ea40a3e00 2024-09-17T15:46:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e9c001a00 con 0x7f2ea413f5b0 2024-09-17T15:46:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): dump: 2024-09-17T15:46:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e980018a0 con 0x7f2ea40a3e00 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:46:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): dump: 2024-09-17T15:46:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.519 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1976407561 0 0) 0x7f2e90003520 con 0x7f2ea40a09b0 2024-09-17T15:46:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ea4073d80 con 0x7f2ea40a09b0 2024-09-17T15:46:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e94001c40 con 0x7f2ea40a09b0 2024-09-17T15:46:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient(hunting): dump: 2024-09-17T15:46:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.522 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1782225688 0 0) 0x7f2ea40a69c0 con 0x7f2ea413f5b0 2024-09-17T15:46:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 >> v1:172.21.15.64:6793/0 conn(0x7f2ea40a09b0 legacy=0x7f2ea4137aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f9d59e60640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 >> v1:172.21.15.100:6790/0 conn(0x7f2ea40a3e00 legacy=0x7f2ea41d2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 monclient: found mon.b 2024-09-17T15:46:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea40734e0 con 0x7f2ea413f5b0 2024-09-17T15:46:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f9d59e60640 10 monclient: _add_conns ranks=[8,1,7,4,5,0,2,6,3] 2024-09-17T15:46:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2e8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2ea9f5d640 5 monclient: authenticate success, global_id 4277 2024-09-17T15:46:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2ea9f5d640 10 monclient: _renew_subs 2024-09-17T15:46:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2ea9f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/600953204 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea4072580 con 0x7f2ea413f5b0 2024-09-17T15:46:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2ea9f5d640 10 monclient: _renew_subs 2024-09-17T15:46:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2ea9f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/600953204 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2ea4074790 con 0x7f2ea413f5b0 2024-09-17T15:46:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f9d59e60640 10 monclient(hunting): picked mon.noname-e con 0x7f9d540a3f50 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f9d59e60640 10 monclient(hunting): picked mon.noname-f con 0x7f9d540a01e0 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f100c27f640 1 Processor -- start 2024-09-17T15:46:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f100c27f640 1 -- start start 2024-09-17T15:46:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f100c27f640 10 monclient: init 2024-09-17T15:46:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.442+0000 7f9d59e60640 10 monclient(hunting): picked mon.noname-i con 0x7f9d540a0670 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d59e60640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d540a4a20 con 0x7f9d540a01e0 2024-09-17T15:46:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d537fe640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f9d540a3f50 0x7f9d5409fad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38598/0 (socket says 172.21.15.64:38598) 2024-09-17T15:46:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2e9c001500 con 0x7f2ea413f5b0 2024-09-17T15:46:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f2e8bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f2ea1ffb640 4 set_mon_vals no callback set 2024-09-17T15:46:05.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d537fe640 1 -- 172.21.15.64:0/1771158498 learned_addr learned my addr 172.21.15.64:0/1771158498 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d52ffd640 1 --1- 172.21.15.64:0/1771158498 >> v1:172.21.15.100:6789/0 conn(0x7f9d540a01e0 0x7f9d5413b650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.64:41154/0 (socket says 172.21.15.64:41154) 2024-09-17T15:46:05.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d59e60640 1 -- 172.21.15.64:0/1771158498 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d5409d720 con 0x7f9d540a0670 2024-09-17T15:46:05.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f100c27f640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f100c27f640 10 monclient: _add_conns ranks=[3,5,8,7,0,4,6,1,2] 2024-09-17T15:46:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f100c27f640 10 monclient(hunting): picked mon.noname-g con 0x7f10040a36b0 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f100c27f640 10 monclient(hunting): picked mon.noname-h con 0x7f10040a41a0 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f100c27f640 10 monclient(hunting): picked mon.noname-e con 0x7f100413c7c0 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f100c27f640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f100409d3b0 con 0x7f10040a36b0 2024-09-17T15:46:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f100c27f640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f100409e450 con 0x7f10040a41a0 2024-09-17T15:46:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f100c27f640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f100409e730 con 0x7f100413c7c0 2024-09-17T15:46:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f100c27f640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f100c27f640 10 monclient(hunting): authenticate will time out at 763.118782s 2024-09-17T15:46:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d527fc640 1 -- 172.21.15.64:0/1771158498 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1292496870 0 0) 0x7f9d540a4a20 con 0x7f9d540a01e0 2024-09-17T15:46:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d59e60640 1 -- 172.21.15.64:0/1771158498 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d540a4c20 con 0x7f9d540a3f50 2024-09-17T15:46:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d59e60640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d59e60640 10 monclient(hunting): authenticate will time out at 763.118897s 2024-09-17T15:46:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f100a01b640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f10040a36b0 0x7f10040a3a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:47730/0 (socket says 172.21.15.64:47730) 2024-09-17T15:46:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f100a01b640 1 -- 172.21.15.64:0/1436345918 learned_addr learned my addr 172.21.15.64:0/1436345918 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f29d8011380 con 0x7f29dc0a54b0 2024-09-17T15:46:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f29d804e5d0 con 0x7f29dc0a54b0 2024-09-17T15:46:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 --> v1:172.21.15.100:6804/2791828047 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29c004bbb0 con 0x7f29c0047f00 2024-09-17T15:46:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f29d804e9d0 con 0x7f29dc0a54b0 2024-09-17T15:46:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f29d1ffb640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-17T15:46:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d527fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d527fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d527fc640 10 monclient(hunting): my global_id is 4339 2024-09-17T15:46:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.443+0000 7f9d527fc640 1 -- 172.21.15.64:0/1771158498 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d3c003520 con 0x7f9d540a01e0 2024-09-17T15:46:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e9c003f60 con 0x7f2ea413f5b0 2024-09-17T15:46:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f2e8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f2e8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f2e8bfff640 10 monclient: dump: 2024-09-17T15:46:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f2ea9f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/600953204 --> v1:172.21.15.64:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f2e84000f80 con 0x7f2ea413f5b0 2024-09-17T15:46:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 1 -- 172.21.15.64:0/1771158498 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3401833345 0 0) 0x7f9d5409d720 con 0x7f9d540a0670 2024-09-17T15:46:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient(hunting): my global_id is 4300 2024-09-17T15:46:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 1 -- 172.21.15.64:0/1771158498 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d540a4a20 con 0x7f9d540a0670 2024-09-17T15:46:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f29e1a63640 10 monclient: get_auth_request con 0x7f29c0047f00 auth_method 0 2024-09-17T15:46:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f29e0a61640 10 monclient: get_auth_request con 0x7f29c00413a0 auth_method 0 2024-09-17T15:46:05.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 1 -- 172.21.15.64:0/1771158498 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1132399812 0 0) 0x7f9d540a4c20 con 0x7f9d540a3f50 2024-09-17T15:46:05.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient(hunting): my global_id is 4292 2024-09-17T15:46:05.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 1 -- 172.21.15.64:0/1771158498 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d5409d720 con 0x7f9d540a3f50 2024-09-17T15:46:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f2e9c004180 con 0x7f2ea413f5b0 2024-09-17T15:46:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 1 -- 172.21.15.64:0/1771158498 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2045729199 0 0) 0x7f9d540a4a20 con 0x7f9d540a0670 2024-09-17T15:46:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 1 -- 172.21.15.64:0/1436345918 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 290630826 0 0) 0x7f100409d3b0 con 0x7f10040a36b0 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient(hunting): my global_id is 4341 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 1 -- 172.21.15.64:0/1436345918 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ff0003520 con 0x7f10040a36b0 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 1 -- 172.21.15.64:0/1436345918 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2938497890 0 0) 0x7f100409e730 con 0x7f100413c7c0 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient(hunting): my global_id is 4301 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 1 -- 172.21.15.64:0/1436345918 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f100409d3b0 con 0x7f100413c7c0 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 1 -- 172.21.15.64:0/1436345918 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3081233578 0 0) 0x7f100409e450 con 0x7f10040a41a0 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient(hunting): my global_id is 4271 2024-09-17T15:46:05.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 1 -- 172.21.15.64:0/1436345918 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f100409e730 con 0x7f10040a41a0 2024-09-17T15:46:05.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 1 -- 172.21.15.64:0/1771158498 >> v1:172.21.15.64:6793/0 conn(0x7f9d540a3f50 legacy=0x7f9d5409fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 1 -- 172.21.15.64:0/1436345918 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4129053738 0 0) 0x7f0ff0003520 con 0x7f10040a36b0 2024-09-17T15:46:05.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 1 -- 172.21.15.64:0/1436345918 >> v1:172.21.15.64:6793/0 conn(0x7f100413c7c0 legacy=0x7f100413ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 1 -- 172.21.15.64:0/1771158498 >> v1:172.21.15.100:6789/0 conn(0x7f9d540a01e0 legacy=0x7f9d5413b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 1 -- 172.21.15.64:0/1436345918 >> v1:172.21.15.100:6791/0 conn(0x7f10040a41a0 legacy=0x7f100406d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 1 monclient: found mon.noname-g 2024-09-17T15:46:05.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.100:6790/0 2024-09-17T15:46:05.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 1 -- 172.21.15.64:0/1436345918 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10040a20f0 con 0x7f10040a36b0 2024-09-17T15:46:05.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 1 monclient: found mon.noname-i 2024-09-17T15:46:05.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.100:6792/0 2024-09-17T15:46:05.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 1 -- 172.21.15.64:0/1771158498 --> v1:172.21.15.100:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5409b560 con 0x7f9d540a0670 2024-09-17T15:46:05.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 1 -- 172.21.15.64:0/1436345918 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1000001bb0 con 0x7f10040a36b0 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f1009019640 10 monclient: dump: 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.542 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.543 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.543 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.543 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.543 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.543 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.543 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 1 -- 172.21.15.64:0/1771158498 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9d4c001b90 con 0x7f9d540a0670 2024-09-17T15:46:05.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d59e60640 5 monclient: authenticate success, global_id 4300 2024-09-17T15:46:05.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-17T15:46:05.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient: dump: 2024-09-17T15:46:05.543 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.544 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.545 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d59e60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f2e9c04da00 con 0x7f2ea413f5b0 2024-09-17T15:46:05.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 --> v1:172.21.15.64:6800/3889666033 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e90053ef0 con 0x7f2e90050240 2024-09-17T15:46:05.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f2e9c04de70 con 0x7f2ea413f5b0 2024-09-17T15:46:05.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f2e8bfff640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-17T15:46:05.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f100c27f640 5 monclient: authenticate success, global_id 4341 2024-09-17T15:46:05.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f100c27f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f2ea37fe640 10 monclient: get_auth_request con 0x7f2e90049640 auth_method 0 2024-09-17T15:46:05.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 1 -- 172.21.15.64:0/1771158498 <== mon.7 v1:172.21.15.100:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d4c001d30 con 0x7f9d540a0670 2024-09-17T15:46:05.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d59e60640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 1 -- 172.21.15.64:0/1771158498 <== mon.7 v1:172.21.15.100:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9d4c003cd0 con 0x7f9d540a0670 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.444+0000 7f9d59e60640 4 set_mon_vals no callback set 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f9d527fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f9d527fc640 10 monclient: dump: 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.547 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.548 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.548 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.548 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.548 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.548 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.548 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.548 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.548 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f1009019640 1 -- 172.21.15.64:0/1436345918 <== mon.3 v1:172.21.15.100:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1000001d50 con 0x7f10040a36b0 2024-09-17T15:46:05.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f1009019640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f1009019640 1 -- 172.21.15.64:0/1436345918 <== mon.3 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1000004f60 con 0x7f10040a36b0 2024-09-17T15:46:05.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f1009019640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f1009019640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:46:05.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f1009019640 10 monclient: dump: 2024-09-17T15:46:05.550 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.550 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.550 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.550 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.550 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.550 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.550 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.551 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.551 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.551 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.551 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.551 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.551 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.551 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.551 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.551 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f100c27f640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:05.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f2ea2ffd640 10 monclient: get_auth_request con 0x7f2e90050240 auth_method 0 2024-09-17T15:46:05.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f100c27f640 4 set_mon_vals no callback set 2024-09-17T15:46:05.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== osd.3 v1:172.21.15.100:6804/2791828047 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f29c004bbb0 con 0x7f29c0047f00 2024-09-17T15:46:05.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f9d59e60640 10 monclient: shutdown 2024-09-17T15:46:05.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f9d59e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f9d59e60640 1 -- 172.21.15.64:0/1771158498 >> v1:172.21.15.100:6792/0 conn(0x7f9d540a0670 legacy=0x7f9d5413ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f9d59e60640 1 -- 172.21.15.64:0/1771158498 shutdown_connections 2024-09-17T15:46:05.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f9d59e60640 1 -- 172.21.15.64:0/1771158498 >> 172.21.15.64:0/1771158498 conn(0x7f9d540a53c0 msgr2=0x7f9d540a57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f9d59e60640 1 -- 172.21.15.64:0/1771158498 shutdown_connections 2024-09-17T15:46:05.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.445+0000 7f9d59e60640 1 -- 172.21.15.64:0/1771158498 wait complete. 2024-09-17T15:46:05.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.446+0000 7f100c27f640 10 monclient: shutdown 2024-09-17T15:46:05.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.446+0000 7f100c27f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.446+0000 7f100c27f640 1 -- 172.21.15.64:0/1436345918 >> v1:172.21.15.100:6790/0 conn(0x7f10040a36b0 legacy=0x7f10040a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.446+0000 7f100c27f640 1 -- 172.21.15.64:0/1436345918 shutdown_connections 2024-09-17T15:46:05.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.446+0000 7f100c27f640 1 -- 172.21.15.64:0/1436345918 >> 172.21.15.64:0/1436345918 conn(0x7f10040a82d0 msgr2=0x7f10040a47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.446+0000 7f100c27f640 1 -- 172.21.15.64:0/1436345918 shutdown_connections 2024-09-17T15:46:05.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.446+0000 7f100c27f640 1 -- 172.21.15.64:0/1436345918 wait complete. 2024-09-17T15:46:05.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.446+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== osd.1 v1:172.21.15.64:6800/3889666033 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e90053ef0 con 0x7f2e90050240 2024-09-17T15:46:05.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.446+0000 7f9d59e60640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.446+0000 7f9d59e60640 10 monclient: monmap: 2024-09-17T15:46:05.555 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.555 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:05.555 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:05.555 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:05.555 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.555 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:05.555 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.555 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.555 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.556 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.556 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.556 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.556 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.556 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.556 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.556 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.446+0000 7f100c27f640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.446+0000 7f9d59e60640 1 Processor -- start 2024-09-17T15:46:05.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.446+0000 7f100c27f640 10 monclient: monmap: 2024-09-17T15:46:05.557 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.557 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:05.557 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:05.557 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:05.557 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.557 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:05.558 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.558 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.558 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.558 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.558 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.558 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.558 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.558 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.558 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.558 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d59e60640 1 -- start start 2024-09-17T15:46:05.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d59e60640 10 monclient: init 2024-09-17T15:46:05.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f100c27f640 1 Processor -- start 2024-09-17T15:46:05.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f100c27f640 1 -- start start 2024-09-17T15:46:05.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f100c27f640 10 monclient: init 2024-09-17T15:46:05.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d59e60640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d59e60640 10 monclient: _add_conns ranks=[3,7,8,5,1,4,2,6,0] 2024-09-17T15:46:05.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d59e60640 10 monclient(hunting): picked mon.noname-g con 0x7f9d540a01e0 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:05.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d59e60640 10 monclient(hunting): picked mon.noname-i con 0x7f9d540a0670 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:05.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d59e60640 10 monclient(hunting): picked mon.noname-e con 0x7f9d540a3f50 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:05.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d59e60640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d54096900 con 0x7f9d540a01e0 2024-09-17T15:46:05.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d59e60640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d540a3040 con 0x7f9d540a0670 2024-09-17T15:46:05.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d59e60640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d540742e0 con 0x7f9d540a3f50 2024-09-17T15:46:05.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d537fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f9d540a01e0 0x7f9d5409a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:47746/0 (socket says 172.21.15.64:47746) 2024-09-17T15:46:05.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d537fe640 1 -- 172.21.15.64:0/13162317 learned_addr learned my addr 172.21.15.64:0/13162317 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d59e60640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.447+0000 7f9d59e60640 10 monclient(hunting): authenticate will time out at 763.122787s 2024-09-17T15:46:05.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2022345578 0 0) 0x7f9d54096900 con 0x7f9d540a01e0 2024-09-17T15:46:05.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): my global_id is 4350 2024-09-17T15:46:05.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d40003550 con 0x7f9d540a01e0 2024-09-17T15:46:05.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2779447456 0 0) 0x7f9d540742e0 con 0x7f9d540a3f50 2024-09-17T15:46:05.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): my global_id is 4310 2024-09-17T15:46:05.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d54096900 con 0x7f9d540a3f50 2024-09-17T15:46:05.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3560329300 0 0) 0x7f9d540a3040 con 0x7f9d540a0670 2024-09-17T15:46:05.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f100c27f640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f100c27f640 10 monclient: _add_conns ranks=[2,4,0,3,5,1,6,7,8] 2024-09-17T15:46:05.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f100c27f640 10 monclient(hunting): picked mon.noname-b con 0x7f10040a36b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:05.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f100c27f640 10 monclient(hunting): picked mon.noname-c con 0x7f10040a41a0 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:05.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f100c27f640 10 monclient(hunting): picked mon.noname-a con 0x7f100413c7c0 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:05.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f100c27f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1004137e60 con 0x7f100413c7c0 2024-09-17T15:46:05.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f100c27f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1004136a40 con 0x7f10040a36b0 2024-09-17T15:46:05.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f100c27f640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10040a4c60 con 0x7f10040a41a0 2024-09-17T15:46:05.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f100c27f640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f100c27f640 10 monclient(hunting): authenticate will time out at 763.123168s 2024-09-17T15:46:05.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f100a01b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f10040a36b0 0x7f1004135c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46772/0 (socket says 172.21.15.64:46772) 2024-09-17T15:46:05.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f100a01b640 1 -- 172.21.15.64:0/1856872252 learned_addr learned my addr 172.21.15.64:0/1856872252 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1642933551 0 0) 0x7f10040a4c60 con 0x7f10040a41a0 2024-09-17T15:46:05.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 10 monclient(hunting): my global_id is 4333 2024-09-17T15:46:05.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fe8003520 con 0x7f10040a41a0 2024-09-17T15:46:05.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2005230733 0 0) 0x7f1004137e60 con 0x7f100413c7c0 2024-09-17T15:46:05.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 10 monclient(hunting): my global_id is 4284 2024-09-17T15:46:05.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10040a4c60 con 0x7f100413c7c0 2024-09-17T15:46:05.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 804742588 0 0) 0x7f1004136a40 con 0x7f10040a36b0 2024-09-17T15:46:05.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 10 monclient(hunting): my global_id is 4286 2024-09-17T15:46:05.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1004137e60 con 0x7f10040a36b0 2024-09-17T15:46:05.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3170852138 0 0) 0x7f0fe8003520 con 0x7f10040a41a0 2024-09-17T15:46:05.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1004136a40 con 0x7f10040a41a0 2024-09-17T15:46:05.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0ff4001a90 con 0x7f10040a41a0 2024-09-17T15:46:05.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:05.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 10 monclient(hunting): dump: 2024-09-17T15:46:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.569 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1608369939 0 0) 0x7f10040a4c60 con 0x7f100413c7c0 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fe8003520 con 0x7f100413c7c0 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3597914491 0 0) 0x7f1004137e60 con 0x7f10040a36b0 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10040a4c60 con 0x7f10040a36b0 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0ffc001a50 con 0x7f100413c7c0 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 10 monclient(hunting): dump: 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.570 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.571 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.571 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.571 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.571 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.571 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.571 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.571 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.571 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.571 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.571 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.571 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.572 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.572 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1000001af0 con 0x7f10040a36b0 2024-09-17T15:46:05.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): my global_id is 4309 2024-09-17T15:46:05.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d540742e0 con 0x7f9d540a0670 2024-09-17T15:46:05.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2256504169 0 0) 0x7f9d40003550 con 0x7f9d540a01e0 2024-09-17T15:46:05.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d540a3040 con 0x7f9d540a01e0 2024-09-17T15:46:05.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9d44001ab0 con 0x7f9d540a01e0 2024-09-17T15:46:05.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): dump: 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.573 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3631450083 0 0) 0x7f9d54096900 con 0x7f9d540a3f50 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d40003550 con 0x7f9d540a3f50 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 838349796 0 0) 0x7f9d540a3040 con 0x7f9d540a01e0 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 >> v1:172.21.15.64:6793/0 conn(0x7f9d540a3f50 legacy=0x7f9d541d5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 >> v1:172.21.15.100:6792/0 conn(0x7f9d540a0670 legacy=0x7f9d541d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 monclient: found mon.g 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d54098c60 con 0x7f9d540a01e0 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d59e60640 5 monclient: authenticate success, global_id 4350 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d59e60640 10 monclient: _renew_subs 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d59e60640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.448+0000 7f9d59e60640 1 -- 172.21.15.64:0/13162317 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9d540983c0 con 0x7f9d540a01e0 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f9d59e60640 10 monclient: _renew_subs 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f9d59e60640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f9d59e60640 1 -- 172.21.15.64:0/13162317 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9d5409a8a0 con 0x7f9d540a01e0 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== mon.3 v1:172.21.15.100:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d44001d30 con 0x7f9d540a01e0 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f9d3bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== mon.3 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9d44004dd0 con 0x7f9d540a01e0 2024-09-17T15:46:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f9d3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f9d3bfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f9d3bfff640 10 monclient: dump: 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.576 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.577 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.577 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.577 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.577 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.449+0000 7f9d51ffb640 4 set_mon_vals no callback set 2024-09-17T15:46:05.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:05.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 10 monclient(hunting): dump: 2024-09-17T15:46:05.578 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.578 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.578 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.578 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.578 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.578 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.578 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.578 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.578 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.578 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.578 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.578 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.579 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3320885401 0 0) 0x7f1004136a40 con 0x7f10040a41a0 2024-09-17T15:46:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 >> v1:172.21.15.64:6790/0 conn(0x7f10040a36b0 legacy=0x7f1004135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 >> v1:172.21.15.64:6789/0 conn(0x7f100413c7c0 legacy=0x7f10041d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 1 monclient: found mon.c 2024-09-17T15:46:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10041353b0 con 0x7f10040a41a0 2024-09-17T15:46:05.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f9d59e60640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:46:05.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f9d59e60640 1 -- 172.21.15.64:0/13162317 --> v1:172.21.15.100:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f9d34000f80 con 0x7f9d540a01e0 2024-09-17T15:46:05.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f100c27f640 5 monclient: authenticate success, global_id 4333 2024-09-17T15:46:05.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f100c27f640 10 monclient: _renew_subs 2024-09-17T15:46:05.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f100c27f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:05.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f100c27f640 1 -- 172.21.15.64:0/1856872252 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1004138cd0 con 0x7f10040a41a0 2024-09-17T15:46:05.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f100c27f640 10 monclient: _renew_subs 2024-09-17T15:46:05.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f100c27f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:05.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f100c27f640 1 -- 172.21.15.64:0/1856872252 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f1004137740 con 0x7f10040a41a0 2024-09-17T15:46:05.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== mon.3 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f9d440116b0 con 0x7f9d540a01e0 2024-09-17T15:46:05.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ff40038c0 con 0x7f10040a41a0 2024-09-17T15:46:05.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0ff4001c30 con 0x7f10040a41a0 2024-09-17T15:46:05.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f1008818640 4 set_mon_vals no callback set 2024-09-17T15:46:05.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:05.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f0ffaffd640 10 monclient: dump: 2024-09-17T15:46:05.586 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.586 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.586 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.586 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.586 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.586 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.587 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.587 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.587 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.587 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.587 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.587 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.588 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.588 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.588 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.588 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== mon.3 v1:172.21.15.100:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f9d4404ea10 con 0x7f9d540a01e0 2024-09-17T15:46:05.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.450+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 --> v1:172.21.15.100:6808/4242749853 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d4004ac90 con 0x7f9d40047000 2024-09-17T15:46:05.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.451+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== mon.3 v1:172.21.15.100:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f9d44016d10 con 0x7f9d540a01e0 2024-09-17T15:46:05.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.451+0000 7f9d3bfff640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-17T15:46:05.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.451+0000 7f9d53fff640 10 monclient: get_auth_request con 0x7f9d40047000 auth_method 0 2024-09-17T15:46:05.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.451+0000 7f9d52ffd640 10 monclient: get_auth_request con 0x7f9d400404c0 auth_method 0 2024-09-17T15:46:05.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.451+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f0ff40104f0 con 0x7f10040a41a0 2024-09-17T15:46:05.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.451+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f0ff404d870 con 0x7f10040a41a0 2024-09-17T15:46:05.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.452+0000 7f100a01b640 10 monclient: get_auth_request con 0x7f0fe8040dd0 auth_method 0 2024-09-17T15:46:05.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.452+0000 7f100c27f640 1 -- 172.21.15.64:0/1856872252 --> v1:172.21.15.64:6804/1825798494 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0fec0053a0 con 0x7f0fec001630 2024-09-17T15:46:05.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.452+0000 7f100a81c640 10 monclient: get_auth_request con 0x7f0fec001630 auth_method 0 2024-09-17T15:46:05.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.452+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== osd.4 v1:172.21.15.100:6808/4242749853 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9d4c00fa50 con 0x7f9d40047000 2024-09-17T15:46:05.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.453+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== osd.2 v1:172.21.15.64:6804/1825798494 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0fec0053a0 con 0x7f0fec001630 2024-09-17T15:46:05.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.462+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/600953204 --> v1:172.21.15.64:6800/3889666033 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2e84002c70 con 0x7f2e90050240 2024-09-17T15:46:05.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.463+0000 7f2e8bfff640 1 -- 172.21.15.64:0/600953204 <== osd.1 v1:172.21.15.64:6800/3889666033 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 191306902) 0x7f2e84002c70 con 0x7f2e90050240 2024-09-17T15:46:05.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.463+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/600953204 >> v1:172.21.15.64:6800/3889666033 conn(0x7f2e90050240 legacy=0x7f2e90052680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.463+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/600953204 >> v1:172.21.15.100:6812/4210963484 conn(0x7f2e90049640 legacy=0x7f2e9004bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.463+0000 7f2ea9f5d640 10 monclient: shutdown 2024-09-17T15:46:05.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.463+0000 7f2ea9f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.463+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/600953204 >> v1:172.21.15.64:6790/0 conn(0x7f2ea413f5b0 legacy=0x7f2ea41d6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.463+0000 7f2ea3fff640 1 -- 172.21.15.64:0/600953204 reap_dead start 2024-09-17T15:46:05.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.463+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/600953204 shutdown_connections 2024-09-17T15:46:05.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.463+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/600953204 >> 172.21.15.64:0/600953204 conn(0x7f2ea40a4ef0 msgr2=0x7f2ea413a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.464+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/600953204 shutdown_connections 2024-09-17T15:46:05.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.464+0000 7f2ea9f5d640 1 -- 172.21.15.64:0/600953204 wait complete. 2024-09-17T15:46:05.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.466+0000 7f9d59e60640 1 -- 172.21.15.64:0/13162317 --> v1:172.21.15.100:6808/4242749853 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9d34002c70 con 0x7f9d40047000 2024-09-17T15:46:05.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.467+0000 7f9d3bfff640 1 -- 172.21.15.64:0/13162317 <== osd.4 v1:172.21.15.100:6808/4242749853 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 258058390) 0x7f9d4c009020 con 0x7f9d40047000 2024-09-17T15:46:05.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.467+0000 7f9d59e60640 1 -- 172.21.15.64:0/13162317 >> v1:172.21.15.100:6808/4242749853 conn(0x7f9d40047000 legacy=0x7f9d40049420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.468+0000 7f9d59e60640 1 -- 172.21.15.64:0/13162317 >> v1:172.21.15.100:6812/4210963484 conn(0x7f9d400404c0 legacy=0x7f9d40042980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.468+0000 7f9d59e60640 10 monclient: shutdown 2024-09-17T15:46:05.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.468+0000 7f9d59e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.468+0000 7f9d59e60640 1 -- 172.21.15.64:0/13162317 >> v1:172.21.15.100:6790/0 conn(0x7f9d540a01e0 legacy=0x7f9d5409a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.468+0000 7f9d53fff640 1 -- 172.21.15.64:0/13162317 reap_dead start 2024-09-17T15:46:05.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.468+0000 7f9d59e60640 1 -- 172.21.15.64:0/13162317 shutdown_connections 2024-09-17T15:46:05.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.468+0000 7f9d59e60640 1 -- 172.21.15.64:0/13162317 >> 172.21.15.64:0/13162317 conn(0x7f9d540a53c0 msgr2=0x7f9d540a7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.468+0000 7f9d59e60640 1 -- 172.21.15.64:0/13162317 shutdown_connections 2024-09-17T15:46:05.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.468+0000 7f9d59e60640 1 -- 172.21.15.64:0/13162317 wait complete. 2024-09-17T15:46:05.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.474+0000 7f29e34c6640 1 -- 172.21.15.64:0/1106961151 --> v1:172.21.15.100:6804/2791828047 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f29dc142890 con 0x7f29c0047f00 2024-09-17T15:46:05.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.474+0000 7f29d1ffb640 1 -- 172.21.15.64:0/1106961151 <== osd.3 v1:172.21.15.100:6804/2791828047 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3325991546) 0x7f29dc142890 con 0x7f29c0047f00 2024-09-17T15:46:05.597 INFO:teuthology.orchestra.run.smithi064.stdout:60129542150 2024-09-17T15:46:05.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T15:46:05.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.475+0000 7f29e34c6640 1 -- 172.21.15.64:0/1106961151 >> v1:172.21.15.100:6804/2791828047 conn(0x7f29c0047f00 legacy=0x7f29c004a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.475+0000 7f29e34c6640 1 -- 172.21.15.64:0/1106961151 >> v1:172.21.15.100:6812/4210963484 conn(0x7f29c00413a0 legacy=0x7f29c0043860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.475+0000 7f29e34c6640 10 monclient: shutdown 2024-09-17T15:46:05.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.475+0000 7f29e34c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.475+0000 7f29e34c6640 1 -- 172.21.15.64:0/1106961151 >> v1:172.21.15.64:6789/0 conn(0x7f29dc0a54b0 legacy=0x7f29dc1372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.475+0000 7f29e1a63640 1 -- 172.21.15.64:0/1106961151 reap_dead start 2024-09-17T15:46:05.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.475+0000 7f29e34c6640 1 -- 172.21.15.64:0/1106961151 shutdown_connections 2024-09-17T15:46:05.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.475+0000 7f29e34c6640 1 -- 172.21.15.64:0/1106961151 >> 172.21.15.64:0/1106961151 conn(0x7f29dc06daa0 msgr2=0x7f29dc0a4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.475+0000 7f29e34c6640 1 -- 172.21.15.64:0/1106961151 shutdown_connections 2024-09-17T15:46:05.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.475+0000 7f29e34c6640 1 -- 172.21.15.64:0/1106961151 wait complete. 2024-09-17T15:46:05.600 INFO:teuthology.orchestra.run.smithi064.stdout:55834574854 2024-09-17T15:46:05.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T15:46:05.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.479+0000 7f100c27f640 1 -- 172.21.15.64:0/1856872252 --> v1:172.21.15.64:6804/1825798494 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0fec007000 con 0x7f0fec001630 2024-09-17T15:46:05.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.479+0000 7f0ffaffd640 1 -- 172.21.15.64:0/1856872252 <== osd.2 v1:172.21.15.64:6804/1825798494 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 4178354069) 0x7f0fec007000 con 0x7f0fec001630 2024-09-17T15:46:05.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.480+0000 7f100c27f640 1 -- 172.21.15.64:0/1856872252 >> v1:172.21.15.64:6804/1825798494 conn(0x7f0fec001630 legacy=0x7f0fec003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.480+0000 7f100c27f640 1 -- 172.21.15.64:0/1856872252 >> v1:172.21.15.100:6812/4210963484 conn(0x7f0fe8040dd0 legacy=0x7f0fe8043290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.480+0000 7f100c27f640 10 monclient: shutdown 2024-09-17T15:46:05.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.480+0000 7f100c27f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.480+0000 7f100c27f640 1 -- 172.21.15.64:0/1856872252 >> v1:172.21.15.64:6791/0 conn(0x7f10040a41a0 legacy=0x7f1004136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.480+0000 7f100a81c640 1 -- 172.21.15.64:0/1856872252 reap_dead start 2024-09-17T15:46:05.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.480+0000 7f100c27f640 1 -- 172.21.15.64:0/1856872252 shutdown_connections 2024-09-17T15:46:05.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.480+0000 7f100c27f640 1 -- 172.21.15.64:0/1856872252 >> 172.21.15.64:0/1856872252 conn(0x7f10040a82d0 msgr2=0x7f100409ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.480+0000 7f100c27f640 1 -- 172.21.15.64:0/1856872252 shutdown_connections 2024-09-17T15:46:05.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.480+0000 7f100c27f640 1 -- 172.21.15.64:0/1856872252 wait complete. 2024-09-17T15:46:05.603 INFO:teuthology.orchestra.run.smithi064.stdout:47244640262 2024-09-17T15:46:05.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T15:46:05.604 INFO:teuthology.orchestra.run.smithi064.stdout:60129542151 2024-09-17T15:46:05.605 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T15:46:05.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.751+0000 7f9af309b640 10 monclient: get_monmap_and_config 2024-09-17T15:46:05.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.751+0000 7f9af309b640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.751+0000 7f9af309b640 10 monclient: monmap: 2024-09-17T15:46:05.753 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.753 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:05.752344+0000 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:05.752344+0000 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.754 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.751+0000 7fd2f3a33640 10 monclient: get_monmap_and_config 2024-09-17T15:46:05.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.751+0000 7fd2f3a33640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.751+0000 7fd2f3a33640 10 monclient: monmap: 2024-09-17T15:46:05.755 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.755 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.755 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:05.752823+0000 2024-09-17T15:46:05.755 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:05.752823+0000 2024-09-17T15:46:05.755 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.756 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.756 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.756 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.756 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.756 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.756 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.756 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.756 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.756 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.756 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.756 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.752+0000 7f9af309b640 1 Processor -- start 2024-09-17T15:46:05.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.752+0000 7fd2f3a33640 1 Processor -- start 2024-09-17T15:46:05.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.752+0000 7f9af309b640 1 -- start start 2024-09-17T15:46:05.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.752+0000 7f9af309b640 10 monclient: init 2024-09-17T15:46:05.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.752+0000 7fd2f3a33640 1 -- start start 2024-09-17T15:46:05.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.752+0000 7fd2f3a33640 10 monclient: init 2024-09-17T15:46:05.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7f9af309b640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7f9af309b640 10 monclient: _add_conns ranks=[7,5,3,8,4,1,0,2,6] 2024-09-17T15:46:05.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7f9af309b640 10 monclient(hunting): picked mon.noname-i con 0x7f9aec0a54b0 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:05.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7f9af309b640 10 monclient(hunting): picked mon.noname-h con 0x7f9aec135c80 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:05.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7f9af309b640 10 monclient(hunting): picked mon.noname-g con 0x7f9aec1360a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:05.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7f9af309b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aec0aacb0 con 0x7f9aec1360a0 2024-09-17T15:46:05.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7f9af309b640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aec09d780 con 0x7f9aec135c80 2024-09-17T15:46:05.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7fd2f3a33640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7f9af309b640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aec09da60 con 0x7f9aec0a54b0 2024-09-17T15:46:05.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7f9af309b640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7fd2f3a33640 10 monclient: _add_conns ranks=[8,7,1,2,5,6,0,3,4] 2024-09-17T15:46:05.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7fd2f3a33640 10 monclient(hunting): picked mon.noname-e con 0x7fd2ec0688b0 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:05.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7f9af309b640 10 monclient(hunting): authenticate will time out at 763.428561s 2024-09-17T15:46:05.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7fd2f3a33640 10 monclient(hunting): picked mon.noname-i con 0x7fd2ec0a36e0 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:05.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7f9aebfff640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7f9aec135c80 0x7f9aec13df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:36746/0 (socket says 172.21.15.64:36746) 2024-09-17T15:46:05.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7f9aebfff640 1 -- 172.21.15.64:0/1341470471 learned_addr learned my addr 172.21.15.64:0/1341470471 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7fd2f3a33640 10 monclient(hunting): picked mon.noname-f con 0x7fd2ec0a3ae0 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7fd2f17cf640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fd2ec0688b0 0x7fd2ec0a2fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38624/0 (socket says 172.21.15.64:38624) 2024-09-17T15:46:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7fd2f17cf640 1 -- 172.21.15.64:0/2574620497 learned_addr learned my addr 172.21.15.64:0/2574620497 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7fd2f3a33640 1 -- 172.21.15.64:0/2574620497 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2ec0a0890 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7fd2f3a33640 1 -- 172.21.15.64:0/2574620497 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2ec0a0540 con 0x7fd2ec0a36e0 2024-09-17T15:46:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7fd2f3a33640 1 -- 172.21.15.64:0/2574620497 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2ec09fad0 con 0x7fd2ec0688b0 2024-09-17T15:46:05.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7fd2f3a33640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 1 -- 172.21.15.64:0/1341470471 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1932727151 0 0) 0x7f9aec09d780 con 0x7f9aec135c80 2024-09-17T15:46:05.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient(hunting): my global_id is 4280 2024-09-17T15:46:05.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 1 -- 172.21.15.64:0/1341470471 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9adc003520 con 0x7f9aec135c80 2024-09-17T15:46:05.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.753+0000 7fd2f3a33640 10 monclient(hunting): authenticate will time out at 763.428956s 2024-09-17T15:46:05.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 -- 172.21.15.64:0/2574620497 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4095075736 0 0) 0x7fd2ec0a0540 con 0x7fd2ec0a36e0 2024-09-17T15:46:05.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient(hunting): my global_id is 4327 2024-09-17T15:46:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 -- 172.21.15.64:0/2574620497 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2d8003520 con 0x7fd2ec0a36e0 2024-09-17T15:46:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 -- 172.21.15.64:0/2574620497 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2921948598 0 0) 0x7fd2ec0a0890 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient(hunting): my global_id is 4348 2024-09-17T15:46:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 -- 172.21.15.64:0/2574620497 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2ec0a0540 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 -- 172.21.15.64:0/2574620497 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3396028609 0 0) 0x7fd2ec09fad0 con 0x7fd2ec0688b0 2024-09-17T15:46:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient(hunting): my global_id is 4319 2024-09-17T15:46:05.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 -- 172.21.15.64:0/2574620497 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2ec0a0890 con 0x7fd2ec0688b0 2024-09-17T15:46:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 1 -- 172.21.15.64:0/1341470471 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3034647866 0 0) 0x7f9aec09da60 con 0x7f9aec0a54b0 2024-09-17T15:46:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient(hunting): my global_id is 4318 2024-09-17T15:46:05.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 1 -- 172.21.15.64:0/1341470471 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9aec09d780 con 0x7f9aec0a54b0 2024-09-17T15:46:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 1 -- 172.21.15.64:0/1341470471 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3232465982 0 0) 0x7f9aec0aacb0 con 0x7f9aec1360a0 2024-09-17T15:46:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient(hunting): my global_id is 4359 2024-09-17T15:46:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 1 -- 172.21.15.64:0/1341470471 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9aec09da60 con 0x7f9aec1360a0 2024-09-17T15:46:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 1 -- 172.21.15.64:0/1341470471 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1857890957 0 0) 0x7f9adc003520 con 0x7f9aec135c80 2024-09-17T15:46:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 1 -- 172.21.15.64:0/1341470471 >> v1:172.21.15.100:6792/0 conn(0x7f9aec0a54b0 legacy=0x7f9aec0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 -- 172.21.15.64:0/2574620497 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 972651193 0 0) 0x7fd2ec0a0890 con 0x7fd2ec0688b0 2024-09-17T15:46:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 1 -- 172.21.15.64:0/1341470471 >> v1:172.21.15.100:6790/0 conn(0x7f9aec1360a0 legacy=0x7f9aec141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 -- 172.21.15.64:0/2574620497 >> v1:172.21.15.100:6792/0 conn(0x7fd2ec0a36e0 legacy=0x7fd2ec13e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 -- 172.21.15.64:0/2574620497 >> v1:172.21.15.100:6789/0 conn(0x7fd2ec0a3ae0 legacy=0x7fd2ec141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 monclient: found mon.noname-e 2024-09-17T15:46:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:46:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 -- 172.21.15.64:0/2574620497 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ec0a0260 con 0x7fd2ec0688b0 2024-09-17T15:46:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 -- 172.21.15.64:0/2574620497 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd2e8001600 con 0x7fd2ec0688b0 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient: dump: 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.770 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.771 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.771 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.771 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.771 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.771 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.771 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.771 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2f3a33640 5 monclient: authenticate success, global_id 4319 2024-09-17T15:46:05.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2f3a33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 1 monclient: found mon.noname-h 2024-09-17T15:46:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.100:6791/0 2024-09-17T15:46:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 1 -- 172.21.15.64:0/1341470471 --> v1:172.21.15.100:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aec09dd40 con 0x7f9aec135c80 2024-09-17T15:46:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 -- 172.21.15.64:0/2574620497 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd2e80017a0 con 0x7fd2ec0688b0 2024-09-17T15:46:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 1 -- 172.21.15.64:0/2574620497 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd2e80048a0 con 0x7fd2ec0688b0 2024-09-17T15:46:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2e3fff640 10 monclient: dump: 2024-09-17T15:46:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.773 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.774 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.775 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 1 -- 172.21.15.64:0/1341470471 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9ad8001830 con 0x7f9aec135c80 2024-09-17T15:46:05.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-17T15:46:05.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 10 monclient: dump: 2024-09-17T15:46:05.775 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.775 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.775 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.775 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.775 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.775 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.776 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.776 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.776 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.776 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.776 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.776 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.776 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.776 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.776 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.776 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9af309b640 5 monclient: authenticate success, global_id 4280 2024-09-17T15:46:05.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9af309b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2f3a33640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:05.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7fd2f3a33640 4 set_mon_vals no callback set 2024-09-17T15:46:05.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.754+0000 7f9aeb7fe640 1 -- 172.21.15.64:0/1341470471 <== mon.5 v1:172.21.15.100:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9ad8001a50 con 0x7f9aec135c80 2024-09-17T15:46:05.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.755+0000 7f9aeb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.755+0000 7f9af309b640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.755+0000 7f9af309b640 4 set_mon_vals no callback set 2024-09-17T15:46:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.755+0000 7f9aeb7fe640 1 -- 172.21.15.64:0/1341470471 <== mon.5 v1:172.21.15.100:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9ad8004be0 con 0x7f9aec135c80 2024-09-17T15:46:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.755+0000 7f9aeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.755+0000 7f9aeb7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:46:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.755+0000 7f9aeb7fe640 10 monclient: dump: 2024-09-17T15:46:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.778 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.779 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.779 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.755+0000 7fd2f3a33640 10 monclient: shutdown 2024-09-17T15:46:05.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.755+0000 7fd2f3a33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.755+0000 7fd2f3a33640 1 -- 172.21.15.64:0/2574620497 >> v1:172.21.15.64:6793/0 conn(0x7fd2ec0688b0 legacy=0x7fd2ec0a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.756+0000 7f9af309b640 10 monclient: shutdown 2024-09-17T15:46:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.756+0000 7f9af309b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.756+0000 7f9af309b640 1 -- 172.21.15.64:0/1341470471 >> v1:172.21.15.100:6791/0 conn(0x7f9aec135c80 legacy=0x7f9aec13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.756+0000 7fd2f3a33640 1 -- 172.21.15.64:0/2574620497 shutdown_connections 2024-09-17T15:46:05.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.756+0000 7fd2f3a33640 1 -- 172.21.15.64:0/2574620497 >> 172.21.15.64:0/2574620497 conn(0x7fd2ec0a87b0 msgr2=0x7fd2ec0a8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.756+0000 7f9af309b640 1 -- 172.21.15.64:0/1341470471 shutdown_connections 2024-09-17T15:46:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.756+0000 7f9af309b640 1 -- 172.21.15.64:0/1341470471 >> 172.21.15.64:0/1341470471 conn(0x7f9aec06daa0 msgr2=0x7f9aec0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.756+0000 7f9af309b640 1 -- 172.21.15.64:0/1341470471 shutdown_connections 2024-09-17T15:46:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.756+0000 7f9af309b640 1 -- 172.21.15.64:0/1341470471 wait complete. 2024-09-17T15:46:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.756+0000 7fd2f3a33640 1 -- 172.21.15.64:0/2574620497 shutdown_connections 2024-09-17T15:46:05.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.756+0000 7fd2f3a33640 1 -- 172.21.15.64:0/2574620497 wait complete. 2024-09-17T15:46:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.756+0000 7f9af309b640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.757+0000 7f9af309b640 10 monclient: monmap: 2024-09-17T15:46:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.783 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.784 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.757+0000 7fd2f3a33640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.757+0000 7fd2f3a33640 10 monclient: monmap: 2024-09-17T15:46:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.785 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.786 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.757+0000 7f9af309b640 1 Processor -- start 2024-09-17T15:46:05.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.757+0000 7f9af309b640 1 -- start start 2024-09-17T15:46:05.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.757+0000 7fd2f3a33640 1 Processor -- start 2024-09-17T15:46:05.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.757+0000 7f9af309b640 10 monclient: init 2024-09-17T15:46:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.757+0000 7fd2f3a33640 1 -- start start 2024-09-17T15:46:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.757+0000 7fd2f3a33640 10 monclient: init 2024-09-17T15:46:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7fd2f3a33640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7fd2f3a33640 10 monclient: _add_conns ranks=[4,2,5,7,1,3,8,6,0] 2024-09-17T15:46:05.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7fd2f3a33640 10 monclient(hunting): picked mon.noname-c con 0x7fd2ec0688b0 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7f9af309b640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7fd2f3a33640 10 monclient(hunting): picked mon.noname-b con 0x7fd2ec0a36e0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7fd2f17cf640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fd2ec0688b0 0x7fd2ec1cdd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:36766/0 (socket says 172.21.15.64:36766) 2024-09-17T15:46:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7fd2f17cf640 1 -- 172.21.15.64:0/3552820437 learned_addr learned my addr 172.21.15.64:0/3552820437 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7fd2f3a33640 10 monclient(hunting): picked mon.noname-h con 0x7fd2ec0a3ae0 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7fd2f3a33640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2ec071850 con 0x7fd2ec0a36e0 2024-09-17T15:46:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7fd2f3a33640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2ec0726a0 con 0x7fd2ec0688b0 2024-09-17T15:46:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7fd2f3a33640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2ec071e00 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7fd2f3a33640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7fd2f3a33640 10 monclient(hunting): authenticate will time out at 763.433873s 2024-09-17T15:46:05.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7f9af309b640 10 monclient: _add_conns ranks=[4,6,3,8,1,5,2,7,0] 2024-09-17T15:46:05.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7f9af309b640 10 monclient(hunting): picked mon.noname-c con 0x7f9aec0a54b0 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:05.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7f9aebfff640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f9aec135c80 0x7f9aec1d71e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:49450/0 (socket says 172.21.15.64:49450) 2024-09-17T15:46:05.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7f9aebfff640 1 -- 172.21.15.64:0/1978790322 learned_addr learned my addr 172.21.15.64:0/1978790322 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7f9af309b640 10 monclient(hunting): picked mon.noname-d con 0x7f9aec135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:05.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7f9af309b640 10 monclient(hunting): picked mon.noname-g con 0x7f9aec1360a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:05.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 827420573 0 0) 0x7fd2ec0726a0 con 0x7fd2ec0688b0 2024-09-17T15:46:05.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 10 monclient(hunting): my global_id is 4342 2024-09-17T15:46:05.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2d0003520 con 0x7fd2ec0688b0 2024-09-17T15:46:05.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543811960 0 0) 0x7fd2ec071850 con 0x7fd2ec0a36e0 2024-09-17T15:46:05.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 10 monclient(hunting): my global_id is 4295 2024-09-17T15:46:05.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2ec0726a0 con 0x7fd2ec0a36e0 2024-09-17T15:46:05.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 926741143 0 0) 0x7fd2ec071e00 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 10 monclient(hunting): my global_id is 4289 2024-09-17T15:46:05.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2ec071850 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7f9af309b640 1 -- 172.21.15.64:0/1978790322 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aec135260 con 0x7f9aec1360a0 2024-09-17T15:46:05.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7f9af309b640 1 -- 172.21.15.64:0/1978790322 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aec1325b0 con 0x7f9aec0a54b0 2024-09-17T15:46:05.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.758+0000 7f9af309b640 1 -- 172.21.15.64:0/1978790322 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aec140ea0 con 0x7f9aec135c80 2024-09-17T15:46:05.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9af309b640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9af309b640 10 monclient(hunting): authenticate will time out at 763.434073s 2024-09-17T15:46:05.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3452588818 0 0) 0x7f9aec140ea0 con 0x7f9aec135c80 2024-09-17T15:46:05.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient(hunting): my global_id is 4272 2024-09-17T15:46:05.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ad0003520 con 0x7f9aec135c80 2024-09-17T15:46:05.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3293810965 0 0) 0x7f9aec1325b0 con 0x7f9aec0a54b0 2024-09-17T15:46:05.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3304050171 0 0) 0x7fd2ec0726a0 con 0x7fd2ec0a36e0 2024-09-17T15:46:05.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2ec071e00 con 0x7fd2ec0a36e0 2024-09-17T15:46:05.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4238738572 0 0) 0x7fd2ec071850 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient(hunting): my global_id is 4351 2024-09-17T15:46:05.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9aec140ea0 con 0x7f9aec0a54b0 2024-09-17T15:46:05.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1890359855 0 0) 0x7f9ad0003520 con 0x7f9aec135c80 2024-09-17T15:46:05.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9aec1325b0 con 0x7f9aec135c80 2024-09-17T15:46:05.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9ad8004b20 con 0x7f9aec135c80 2024-09-17T15:46:05.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:46:05.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient(hunting): dump: 2024-09-17T15:46:05.797 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.797 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.798 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.798 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.798 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.798 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.798 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.798 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.798 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.798 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.799 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.799 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.799 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.799 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.799 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.799 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:05.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 572377642 0 0) 0x7f9aec1325b0 con 0x7f9aec135c80 2024-09-17T15:46:05.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 >> v1:172.21.15.64:6791/0 conn(0x7f9aec0a54b0 legacy=0x7f9aec1d12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 >> v1:172.21.15.100:6790/0 conn(0x7f9aec1360a0 legacy=0x7f9aec1da910 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 1 monclient: found mon.d 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.759+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aec132050 con 0x7f9aec135c80 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7f9ae97fa640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7f9ae97fa640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7f9ae97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7f9af309b640 5 monclient: authenticate success, global_id 4272 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7f9af309b640 10 monclient: _renew_subs 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7f9af309b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7f9af309b640 1 -- 172.21.15.64:0/1978790322 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9aec1d19d0 con 0x7f9aec135c80 2024-09-17T15:46:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7f9af309b640 10 monclient: _renew_subs 2024-09-17T15:46:05.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7f9af309b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:05.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7f9af309b640 1 -- 172.21.15.64:0/1978790322 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f9aec142320 con 0x7f9aec135c80 2024-09-17T15:46:05.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2ec0726a0 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2885847621 0 0) 0x7fd2d0003520 con 0x7fd2ec0688b0 2024-09-17T15:46:05.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2ec071850 con 0x7fd2ec0688b0 2024-09-17T15:46:05.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd2dc001590 con 0x7fd2ec0a36e0 2024-09-17T15:46:05.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:05.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient(hunting): dump: 2024-09-17T15:46:05.804 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.804 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.804 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.804 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.804 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.804 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.805 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd2e4001bd0 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient(hunting): dump: 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.806 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.807 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.807 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.807 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.807 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.807 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.807 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd2e8001580 con 0x7fd2ec0688b0 2024-09-17T15:46:05.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:05.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient(hunting): dump: 2024-09-17T15:46:05.807 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.807 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.807 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.808 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1099334137 0 0) 0x7fd2ec0726a0 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 >> v1:172.21.15.64:6791/0 conn(0x7fd2ec0688b0 legacy=0x7fd2ec1cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 >> v1:172.21.15.64:6790/0 conn(0x7fd2ec0a36e0 legacy=0x7fd2ec1d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 monclient: found mon.h 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.100:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ec09f3f0 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.760+0000 7fd2e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.761+0000 7fd2f3a33640 5 monclient: authenticate success, global_id 4289 2024-09-17T15:46:05.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.761+0000 7fd2f3a33640 10 monclient: _renew_subs 2024-09-17T15:46:05.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.761+0000 7fd2f3a33640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:46:05.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.761+0000 7fd2f3a33640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.100:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd2ec074f10 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.761+0000 7fd2f3a33640 10 monclient: _renew_subs 2024-09-17T15:46:05.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.761+0000 7fd2f3a33640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:46:05.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.761+0000 7fd2f3a33640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.100:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fd2ec09e710 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.761+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.5 v1:172.21.15.100:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd2e4003990 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.761+0000 7fd2e1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.761+0000 7fd2e37fe640 4 set_mon_vals no callback set 2024-09-17T15:46:05.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.5 v1:172.21.15.100:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd2e4001d70 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7fd2e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7fd2e1ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:46:05.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9ad8004ee0 con 0x7f9aec135c80 2024-09-17T15:46:05.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7f9ae97fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9ad80053d0 con 0x7f9aec135c80 2024-09-17T15:46:05.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7f9ae97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7f9ae97fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:05.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7f9ae97fa640 10 monclient: dump: 2024-09-17T15:46:05.811 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.811 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.812 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.812 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.812 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.812 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.812 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.812 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.812 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.812 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.812 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.812 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.812 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.812 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.813 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.813 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f9ad8011cb0 con 0x7f9aec135c80 2024-09-17T15:46:05.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f9ad804efd0 con 0x7f9aec135c80 2024-09-17T15:46:05.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.763+0000 7f9aeaffd640 4 set_mon_vals no callback set 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7fd2e1ffb640 10 monclient: dump: 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.814 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.815 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.815 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.815 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.5 v1:172.21.15.100:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fd2e4003e80 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.762+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.5 v1:172.21.15.100:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7fd2e404d9e0 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.763+0000 7fd2f17cf640 10 monclient: get_auth_request con 0x7fd2d0040d90 auth_method 0 2024-09-17T15:46:05.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.763+0000 7fd2f3a33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.763+0000 7f9af0e37640 10 monclient: get_auth_request con 0x7f9ad0040470 auth_method 0 2024-09-17T15:46:05.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.763+0000 7fd2f3a33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.763+0000 7f9ad6ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.763+0000 7fd2f3a33640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:46:05.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.763+0000 7fd2f3a33640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.100:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2cc005080 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.764+0000 7f9ad6ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.764+0000 7f9ad6ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:05.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.764+0000 7f9ad6ffd640 1 -- 172.21.15.64:0/1978790322 --> v1:172.21.15.64:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9acc005080 con 0x7f9aec135c80 2024-09-17T15:46:05.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.768+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.5 v1:172.21.15.100:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fd2e4016680 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.768+0000 7fd2e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.768+0000 7fd2e1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:05.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.773+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 <== mon.6 v1:172.21.15.64:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f9ad8017d30 con 0x7f9aec135c80 2024-09-17T15:46:05.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.773+0000 7f9ae97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.773+0000 7f9ae97fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:05.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.785+0000 7f07a74a3640 10 monclient: get_monmap_and_config 2024-09-17T15:46:05.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.785+0000 7f07a74a3640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.786+0000 7f07a74a3640 10 monclient: monmap: 2024-09-17T15:46:05.828 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.828 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.828 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:05.786844+0000 2024-09-17T15:46:05.828 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:05.786844+0000 2024-09-17T15:46:05.828 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.829 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.829 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.829 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.829 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.829 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.829 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.829 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.829 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.829 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.830 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.830 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.786+0000 7f07a74a3640 1 Processor -- start 2024-09-17T15:46:05.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.786+0000 7f07a74a3640 1 -- start start 2024-09-17T15:46:05.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.786+0000 7f07a74a3640 10 monclient: init 2024-09-17T15:46:05.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.787+0000 7f07a74a3640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.787+0000 7f07a74a3640 10 monclient: _add_conns ranks=[6,4,2,0,5,7,8,3,1] 2024-09-17T15:46:05.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.787+0000 7f07a74a3640 10 monclient(hunting): picked mon.noname-d con 0x7f07a00a54b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:05.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.787+0000 7f07a74a3640 10 monclient(hunting): picked mon.noname-c con 0x7f07a0135c80 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:05.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.787+0000 7f07a4a3e640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f07a0135c80 0x7f07a013df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:36788/0 (socket says 172.21.15.64:36788) 2024-09-17T15:46:05.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.787+0000 7f07a4a3e640 1 -- 172.21.15.64:0/505356558 learned_addr learned my addr 172.21.15.64:0/505356558 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.787+0000 7f07a74a3640 10 monclient(hunting): picked mon.noname-b con 0x7f07a01360a0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:05.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.787+0000 7f07a74a3640 1 -- 172.21.15.64:0/505356558 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07a00aacb0 con 0x7f07a01360a0 2024-09-17T15:46:05.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f07a74a3640 1 -- 172.21.15.64:0/505356558 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07a009d780 con 0x7f07a0135c80 2024-09-17T15:46:05.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f07a74a3640 1 -- 172.21.15.64:0/505356558 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07a009da60 con 0x7f07a00a54b0 2024-09-17T15:46:05.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f07a74a3640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f07a74a3640 10 monclient(hunting): authenticate will time out at 763.463183s 2024-09-17T15:46:05.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 1 -- 172.21.15.64:0/505356558 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4290008988 0 0) 0x7f07a009d780 con 0x7f07a0135c80 2024-09-17T15:46:05.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient(hunting): my global_id is 4360 2024-09-17T15:46:05.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 1 -- 172.21.15.64:0/505356558 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f078c003520 con 0x7f07a0135c80 2024-09-17T15:46:05.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 1 -- 172.21.15.64:0/505356558 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1411078293 0 0) 0x7f07a009da60 con 0x7f07a00a54b0 2024-09-17T15:46:05.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient(hunting): my global_id is 4281 2024-09-17T15:46:05.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 1 -- 172.21.15.64:0/505356558 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07a009d780 con 0x7f07a00a54b0 2024-09-17T15:46:05.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 1 -- 172.21.15.64:0/505356558 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3892394314 0 0) 0x7f07a00aacb0 con 0x7f07a01360a0 2024-09-17T15:46:05.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient(hunting): my global_id is 4304 2024-09-17T15:46:05.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 1 -- 172.21.15.64:0/505356558 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07a009da60 con 0x7f07a01360a0 2024-09-17T15:46:05.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 1 -- 172.21.15.64:0/505356558 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3341432893 0 0) 0x7f07a009d780 con 0x7f07a00a54b0 2024-09-17T15:46:05.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 1 -- 172.21.15.64:0/505356558 >> v1:172.21.15.64:6791/0 conn(0x7f07a0135c80 legacy=0x7f07a013df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 1 -- 172.21.15.64:0/505356558 >> v1:172.21.15.64:6790/0 conn(0x7f07a01360a0 legacy=0x7f07a0141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 1 monclient: found mon.noname-d 2024-09-17T15:46:05.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:46:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 1 -- 172.21.15.64:0/505356558 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a009dd40 con 0x7f07a00a54b0 2024-09-17T15:46:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 1 -- 172.21.15.64:0/505356558 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f079c0016c0 con 0x7f07a00a54b0 2024-09-17T15:46:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:46:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.788+0000 7f0797fff640 10 monclient: dump: 2024-09-17T15:46:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.837 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.789+0000 7f07a74a3640 5 monclient: authenticate success, global_id 4281 2024-09-17T15:46:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.789+0000 7f07a74a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.789+0000 7f0797fff640 1 -- 172.21.15.64:0/505356558 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f079c001860 con 0x7f07a00a54b0 2024-09-17T15:46:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.789+0000 7f0797fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.789+0000 7f07a74a3640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.789+0000 7f07a74a3640 4 set_mon_vals no callback set 2024-09-17T15:46:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.789+0000 7f0797fff640 1 -- 172.21.15.64:0/505356558 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f079c0035f0 con 0x7f07a00a54b0 2024-09-17T15:46:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.789+0000 7f0797fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.789+0000 7f0797fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.789+0000 7f0797fff640 10 monclient: dump: 2024-09-17T15:46:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.840 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.840 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.840 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.840 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.840 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.840 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.840 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.840 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.790+0000 7f07a74a3640 10 monclient: shutdown 2024-09-17T15:46:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.790+0000 7f07a74a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.790+0000 7f07a74a3640 1 -- 172.21.15.64:0/505356558 >> v1:172.21.15.64:6792/0 conn(0x7f07a00a54b0 legacy=0x7f07a00a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.790+0000 7f07a74a3640 1 -- 172.21.15.64:0/505356558 shutdown_connections 2024-09-17T15:46:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.790+0000 7f07a74a3640 1 -- 172.21.15.64:0/505356558 >> 172.21.15.64:0/505356558 conn(0x7f07a006daa0 msgr2=0x7f07a00ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.790+0000 7f07a74a3640 1 -- 172.21.15.64:0/505356558 shutdown_connections 2024-09-17T15:46:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.790+0000 7f07a74a3640 1 -- 172.21.15.64:0/505356558 wait complete. 2024-09-17T15:46:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.791+0000 7f07a74a3640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.791+0000 7f07a74a3640 10 monclient: monmap: 2024-09-17T15:46:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.843 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.792+0000 7f07a74a3640 1 Processor -- start 2024-09-17T15:46:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.792+0000 7f07a74a3640 1 -- start start 2024-09-17T15:46:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.792+0000 7f07a74a3640 10 monclient: init 2024-09-17T15:46:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.793+0000 7f07a74a3640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.793+0000 7f07a74a3640 10 monclient: _add_conns ranks=[5,8,1,4,0,3,2,7,6] 2024-09-17T15:46:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.793+0000 7f07a74a3640 10 monclient(hunting): picked mon.noname-h con 0x7f07a00a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.793+0000 7f07a4a3e640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f07a0135c80 0x7f07a01d2710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38626/0 (socket says 172.21.15.64:38626) 2024-09-17T15:46:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.794+0000 7f07a523f640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7f07a00a54b0 0x7f07a00a4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:36754/0 (socket says 172.21.15.64:36754) 2024-09-17T15:46:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.794+0000 7f07a4a3e640 1 -- 172.21.15.64:0/3358980176 learned_addr learned my addr 172.21.15.64:0/3358980176 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.794+0000 7f07a74a3640 10 monclient(hunting): picked mon.noname-e con 0x7f07a0135c80 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.794+0000 7f07a74a3640 10 monclient(hunting): picked mon.noname-f con 0x7f07a01360a0 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.794+0000 7f07a74a3640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07a00a0d60 con 0x7f07a01360a0 2024-09-17T15:46:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.794+0000 7f07a74a3640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07a00a35e0 con 0x7f07a00a54b0 2024-09-17T15:46:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.794+0000 7f07a74a3640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07a0135060 con 0x7f07a0135c80 2024-09-17T15:46:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f07a74a3640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f07a74a3640 10 monclient(hunting): authenticate will time out at 763.470177s 2024-09-17T15:46:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4253770527 0 0) 0x7f07a00a0d60 con 0x7f07a01360a0 2024-09-17T15:46:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): my global_id is 4357 2024-09-17T15:46:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0784003520 con 0x7f07a01360a0 2024-09-17T15:46:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3369334072 0 0) 0x7f07a00a35e0 con 0x7f07a00a54b0 2024-09-17T15:46:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): my global_id is 4298 2024-09-17T15:46:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07a00a0d60 con 0x7f07a00a54b0 2024-09-17T15:46:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 9475576 0 0) 0x7f07a0135060 con 0x7f07a0135c80 2024-09-17T15:46:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): my global_id is 4328 2024-09-17T15:46:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07a00a35e0 con 0x7f07a0135c80 2024-09-17T15:46:05.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 5993433 0 0) 0x7f07a00a35e0 con 0x7f07a0135c80 2024-09-17T15:46:05.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f18fa589640 10 monclient: get_monmap_and_config 2024-09-17T15:46:05.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f18fa589640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f18fa589640 10 monclient: monmap: 2024-09-17T15:46:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:05.796915+0000 2024-09-17T15:46:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:05.796915+0000 2024-09-17T15:46:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.850 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07a0135060 con 0x7f07a0135c80 2024-09-17T15:46:05.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0790001af0 con 0x7f07a0135c80 2024-09-17T15:46:05.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:46:05.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): dump: 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.856 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 642182550 0 0) 0x7f0784003520 con 0x7f07a01360a0 2024-09-17T15:46:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07a00a35e0 con 0x7f07a01360a0 2024-09-17T15:46:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0798001cb0 con 0x7f07a01360a0 2024-09-17T15:46:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.795+0000 7f0795ffb640 10 monclient(hunting): dump: 2024-09-17T15:46:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4137928350 0 0) 0x7f07a00a0d60 con 0x7f07a00a54b0 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0784003520 con 0x7f07a00a54b0 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f079c0035f0 con 0x7f07a00a54b0 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 10 monclient(hunting): dump: 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.859 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2276079302 0 0) 0x7f07a00a35e0 con 0x7f07a01360a0 2024-09-17T15:46:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 >> v1:172.21.15.64:6793/0 conn(0x7f07a0135c80 legacy=0x7f07a01d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 >> v1:172.21.15.100:6791/0 conn(0x7f07a00a54b0 legacy=0x7f07a00a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 1 monclient: found mon.f 2024-09-17T15:46:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:46:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a00a1280 con 0x7f07a01360a0 2024-09-17T15:46:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f07a74a3640 5 monclient: authenticate success, global_id 4357 2024-09-17T15:46:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f07a74a3640 10 monclient: _renew_subs 2024-09-17T15:46:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f07a74a3640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:46:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f07a74a3640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f07a00a3300 con 0x7f07a01360a0 2024-09-17T15:46:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.1 v1:172.21.15.100:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0798003970 con 0x7f07a01360a0 2024-09-17T15:46:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.1 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0798003e60 con 0x7f07a01360a0 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f0795ffb640 10 monclient: dump: 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.862 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.796+0000 7f07977fe640 4 set_mon_vals no callback set 2024-09-17T15:46:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.797+0000 7f780815a640 10 monclient: get_monmap_and_config 2024-09-17T15:46:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.797+0000 7f780815a640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.797+0000 7f780815a640 10 monclient: monmap: 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:05.798284+0000 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:05.798284+0000 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.864 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.797+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.1 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f0798010740 con 0x7f07a01360a0 2024-09-17T15:46:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.797+0000 7f07a74a3640 10 monclient: _renew_subs 2024-09-17T15:46:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.797+0000 7f07a74a3640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:46:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.797+0000 7f07a74a3640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f07a00a1de0 con 0x7f07a01360a0 2024-09-17T15:46:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.798+0000 7f18fa589640 1 Processor -- start 2024-09-17T15:46:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.798+0000 7f780815a640 1 Processor -- start 2024-09-17T15:46:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.798+0000 7f18fa589640 1 -- start start 2024-09-17T15:46:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.798+0000 7f18fa589640 10 monclient: init 2024-09-17T15:46:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.798+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.1 v1:172.21.15.100:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f07980122c0 con 0x7f07a01360a0 2024-09-17T15:46:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.798+0000 7f07a523f640 10 monclient: get_auth_request con 0x7f0784041010 auth_method 0 2024-09-17T15:46:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.798+0000 7f07a74a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.798+0000 7f07a74a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.798+0000 7f07a74a3640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:46:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.798+0000 7f07a74a3640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07a00647b0 con 0x7f07a01360a0 2024-09-17T15:46:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f780815a640 1 -- start start 2024-09-17T15:46:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f18fa589640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f18fa589640 10 monclient: _add_conns ranks=[2,5,8,7,6,1,3,4,0] 2024-09-17T15:46:05.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f780815a640 10 monclient: init 2024-09-17T15:46:05.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f18fa589640 10 monclient(hunting): picked mon.noname-b con 0x7f18f409e830 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f18fa589640 10 monclient(hunting): picked mon.noname-h con 0x7f18f40689d0 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f18f3fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f18f409e830 0x7f18f409ec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46798/0 (socket says 172.21.15.64:46798) 2024-09-17T15:46:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f18f3fff640 1 -- 172.21.15.64:0/4002460380 learned_addr learned my addr 172.21.15.64:0/4002460380 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f18fa589640 10 monclient(hunting): picked mon.noname-e con 0x7f18f4137130 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f18fa589640 1 -- 172.21.15.64:0/4002460380 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18f4069110 con 0x7f18f409e830 2024-09-17T15:46:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f18fa589640 1 -- 172.21.15.64:0/4002460380 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18f409f320 con 0x7f18f40689d0 2024-09-17T15:46:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f780815a640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f780815a640 10 monclient: _add_conns ranks=[7,6,4,5,0,2,1,3,8] 2024-09-17T15:46:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f18fa589640 1 -- 172.21.15.64:0/4002460380 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18f4069380 con 0x7f18f4137130 2024-09-17T15:46:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f780815a640 10 monclient(hunting): picked mon.noname-i con 0x7f78000688b0 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f780815a640 10 monclient(hunting): picked mon.noname-d con 0x7f78000a0c20 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f780815a640 10 monclient(hunting): picked mon.noname-c con 0x7f780013f620 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f780815a640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78000a3ec0 con 0x7f780013f620 2024-09-17T15:46:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f780815a640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78000a0440 con 0x7f78000a0c20 2024-09-17T15:46:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f780815a640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78000a0720 con 0x7f78000688b0 2024-09-17T15:46:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f780815a640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f18f37fe640 1 -- 172.21.15.64:0/4002460380 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2478181065 0 0) 0x7f18f4069110 con 0x7f18f409e830 2024-09-17T15:46:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.799+0000 7f780815a640 10 monclient(hunting): authenticate will time out at 763.474996s 2024-09-17T15:46:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18fa589640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18fa589640 10 monclient(hunting): authenticate will time out at 763.475136s 2024-09-17T15:46:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f78066f7640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f780013f620 0x7f7800141a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:36790/0 (socket says 172.21.15.64:36790) 2024-09-17T15:46:05.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f78066f7640 1 -- 172.21.15.64:0/1613828232 learned_addr learned my addr 172.21.15.64:0/1613828232 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 10 monclient(hunting): my global_id is 4313 2024-09-17T15:46:05.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 1 -- 172.21.15.64:0/4002460380 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18e0003520 con 0x7f18f409e830 2024-09-17T15:46:05.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 1 -- 172.21.15.64:0/1613828232 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 831228413 0 0) 0x7f78000a3ec0 con 0x7f780013f620 2024-09-17T15:46:05.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 1 -- 172.21.15.64:0/4002460380 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1573739422 0 0) 0x7f18f409f320 con 0x7f18f40689d0 2024-09-17T15:46:05.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 10 monclient(hunting): my global_id is 4307 2024-09-17T15:46:05.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 1 -- 172.21.15.64:0/4002460380 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18f4069110 con 0x7f18f40689d0 2024-09-17T15:46:05.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 1 -- 172.21.15.64:0/4002460380 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3245686169 0 0) 0x7f18ec001490 con 0x7f18f4137130 2024-09-17T15:46:05.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient(hunting): my global_id is 4369 2024-09-17T15:46:05.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 1 -- 172.21.15.64:0/1613828232 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77ec003520 con 0x7f780013f620 2024-09-17T15:46:05.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 1 -- 172.21.15.64:0/1613828232 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4001096322 0 0) 0x7f78000a0720 con 0x7f78000688b0 2024-09-17T15:46:05.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient(hunting): my global_id is 4336 2024-09-17T15:46:05.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 10 monclient(hunting): my global_id is 4337 2024-09-17T15:46:05.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 1 -- 172.21.15.64:0/4002460380 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18f409f320 con 0x7f18f4137130 2024-09-17T15:46:05.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 1 -- 172.21.15.64:0/1613828232 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78000a3ec0 con 0x7f78000688b0 2024-09-17T15:46:05.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 1 -- 172.21.15.64:0/1613828232 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 933307685 0 0) 0x7f78000a0440 con 0x7f78000a0c20 2024-09-17T15:46:05.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient(hunting): my global_id is 4290 2024-09-17T15:46:05.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 1 -- 172.21.15.64:0/1613828232 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78000a0720 con 0x7f78000a0c20 2024-09-17T15:46:05.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 1 -- 172.21.15.64:0/4002460380 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3755183855 0 0) 0x7f18e0003520 con 0x7f18f409e830 2024-09-17T15:46:05.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f18f37fe640 1 -- 172.21.15.64:0/4002460380 >> v1:172.21.15.64:6793/0 conn(0x7f18f4137130 legacy=0x7f18f413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 1 -- 172.21.15.64:0/1613828232 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2024139357 0 0) 0x7f77ec003520 con 0x7f780013f620 2024-09-17T15:46:05.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 1 -- 172.21.15.64:0/1613828232 >> v1:172.21.15.100:6792/0 conn(0x7f78000688b0 legacy=0x7f78000a30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 1 -- 172.21.15.64:0/1613828232 >> v1:172.21.15.64:6792/0 conn(0x7f78000a0c20 legacy=0x7f78000a37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 1 monclient: found mon.noname-c 2024-09-17T15:46:05.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:46:05.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 1 -- 172.21.15.64:0/1613828232 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780009f960 con 0x7f780013f620 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 1 -- 172.21.15.64:0/1613828232 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f77f8001bb0 con 0x7f780013f620 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.800+0000 7f7804ef4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f7804ef4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f7804ef4640 10 monclient: dump: 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.878 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.879 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.879 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.879 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.879 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.879 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.879 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.879 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.879 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.879 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f780815a640 5 monclient: authenticate success, global_id 4369 2024-09-17T15:46:05.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f780815a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 1 -- 172.21.15.64:0/4002460380 >> v1:172.21.15.100:6791/0 conn(0x7f18f40689d0 legacy=0x7f18f413bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 1 monclient: found mon.noname-b 2024-09-17T15:46:05.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7fde12eed640 10 monclient: get_monmap_and_config 2024-09-17T15:46:05.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7fde12eed640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 1 -- 172.21.15.64:0/4002460380 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f4069660 con 0x7f18f409e830 2024-09-17T15:46:05.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7fde12eed640 10 monclient: monmap: 2024-09-17T15:46:05.881 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.881 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.881 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:05.802042+0000 2024-09-17T15:46:05.881 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:05.802042+0000 2024-09-17T15:46:05.881 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.881 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.881 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.882 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.882 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.882 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.882 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.882 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.882 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.882 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.882 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.882 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f7804ef4640 1 -- 172.21.15.64:0/1613828232 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f77f8001dd0 con 0x7f780013f620 2024-09-17T15:46:05.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f7804ef4640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f7804ef4640 1 -- 172.21.15.64:0/1613828232 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f77f8004f60 con 0x7f780013f620 2024-09-17T15:46:05.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 1 -- 172.21.15.64:0/4002460380 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f18dc0017b0 con 0x7f18f409e830 2024-09-17T15:46:05.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f780815a640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:05.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f780815a640 4 set_mon_vals no callback set 2024-09-17T15:46:05.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f7804ef4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f7804ef4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:05.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f7804ef4640 10 monclient: dump: 2024-09-17T15:46:05.884 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.884 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.884 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.884 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.885 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.885 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.885 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.885 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.885 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.885 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.885 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.885 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.885 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.885 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.885 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.885 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:05.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 10 monclient: dump: 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.887 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.888 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.888 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 1 -- 172.21.15.64:0/4002460380 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18dc001a10 con 0x7f18f409e830 2024-09-17T15:46:05.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 1 -- 172.21.15.64:0/4002460380 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f18dc004be0 con 0x7f18f409e830 2024-09-17T15:46:05.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:05.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18f37fe640 10 monclient: dump: 2024-09-17T15:46:05.888 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.888 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.888 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18fa589640 5 monclient: authenticate success, global_id 4313 2024-09-17T15:46:05.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18fa589640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:05.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.801+0000 7f18fa589640 4 set_mon_vals no callback set 2024-09-17T15:46:05.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.802+0000 7fde12eed640 1 Processor -- start 2024-09-17T15:46:05.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.802+0000 7f780815a640 10 monclient: shutdown 2024-09-17T15:46:05.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.802+0000 7f780815a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.802+0000 7f780815a640 1 -- 172.21.15.64:0/1613828232 >> v1:172.21.15.64:6791/0 conn(0x7f780013f620 legacy=0x7f7800141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.802+0000 7fde12eed640 1 -- start start 2024-09-17T15:46:05.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.802+0000 7fde12eed640 10 monclient: init 2024-09-17T15:46:05.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.802+0000 7f780815a640 1 -- 172.21.15.64:0/1613828232 shutdown_connections 2024-09-17T15:46:05.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.802+0000 7f780815a640 1 -- 172.21.15.64:0/1613828232 >> 172.21.15.64:0/1613828232 conn(0x7f78000a8640 msgr2=0x7f78000a8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.802+0000 7f780815a640 1 -- 172.21.15.64:0/1613828232 shutdown_connections 2024-09-17T15:46:05.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.802+0000 7f780815a640 1 -- 172.21.15.64:0/1613828232 wait complete. 2024-09-17T15:46:05.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.802+0000 7f18fa589640 10 monclient: shutdown 2024-09-17T15:46:05.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.802+0000 7f18fa589640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.802+0000 7f18fa589640 1 -- 172.21.15.64:0/4002460380 >> v1:172.21.15.64:6790/0 conn(0x7f18f409e830 legacy=0x7f18f409ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde12eed640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde12eed640 10 monclient: _add_conns ranks=[6,2,4,3,5,7,1,0,8] 2024-09-17T15:46:05.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde12eed640 10 monclient(hunting): picked mon.noname-d con 0x7fde0c0a3fe0 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:05.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde12eed640 10 monclient(hunting): picked mon.noname-b con 0x7fde0c0a43c0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:05.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde12eed640 10 monclient(hunting): picked mon.noname-c con 0x7fde0c13bfb0 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:05.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde12eed640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c069df0 con 0x7fde0c0a43c0 2024-09-17T15:46:05.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde12eed640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c0a1080 con 0x7fde0c13bfb0 2024-09-17T15:46:05.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde12eed640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c0a1360 con 0x7fde0c0a3fe0 2024-09-17T15:46:05.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde12eed640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde12eed640 10 monclient(hunting): authenticate will time out at 763.478213s 2024-09-17T15:46:05.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde10c89640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fde0c0a3fe0 0x7fde0c0a2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:49480/0 (socket says 172.21.15.64:49480) 2024-09-17T15:46:05.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde10c89640 1 -- 172.21.15.64:0/1052846474 learned_addr learned my addr 172.21.15.64:0/1052846474 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7f780815a640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7f780815a640 10 monclient: monmap: 2024-09-17T15:46:05.894 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.894 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:05.894 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:05.894 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:05.894 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.895 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:05.895 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.895 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.895 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.895 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.895 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.895 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.895 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.895 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.895 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.895 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7f18fa589640 1 -- 172.21.15.64:0/4002460380 shutdown_connections 2024-09-17T15:46:05.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 1 -- 172.21.15.64:0/1052846474 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 668377128 0 0) 0x7fde0c0a1080 con 0x7fde0c13bfb0 2024-09-17T15:46:05.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 10 monclient(hunting): my global_id is 4378 2024-09-17T15:46:05.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 1 -- 172.21.15.64:0/1052846474 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddfc003520 con 0x7fde0c13bfb0 2024-09-17T15:46:05.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 1 -- 172.21.15.64:0/1052846474 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1679016647 0 0) 0x7fde0c0a1360 con 0x7fde0c0a3fe0 2024-09-17T15:46:05.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 10 monclient(hunting): my global_id is 4299 2024-09-17T15:46:05.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 1 -- 172.21.15.64:0/1052846474 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde0c0a1080 con 0x7fde0c0a3fe0 2024-09-17T15:46:05.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7f18fa589640 1 -- 172.21.15.64:0/4002460380 >> 172.21.15.64:0/4002460380 conn(0x7f18f40a4e80 msgr2=0x7f18f40a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 1 -- 172.21.15.64:0/1052846474 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1500305006 0 0) 0x7fde0c069df0 con 0x7fde0c0a43c0 2024-09-17T15:46:05.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 10 monclient(hunting): my global_id is 4322 2024-09-17T15:46:05.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7fde0b7fe640 1 -- 172.21.15.64:0/1052846474 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde0c0a1360 con 0x7fde0c0a43c0 2024-09-17T15:46:05.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.803+0000 7f780815a640 1 Processor -- start 2024-09-17T15:46:05.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f18fa589640 1 -- 172.21.15.64:0/4002460380 shutdown_connections 2024-09-17T15:46:05.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f18fa589640 1 -- 172.21.15.64:0/4002460380 wait complete. 2024-09-17T15:46:05.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 1 -- 172.21.15.64:0/1052846474 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1954380143 0 0) 0x7fddfc003520 con 0x7fde0c13bfb0 2024-09-17T15:46:05.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 1 -- 172.21.15.64:0/1052846474 >> v1:172.21.15.64:6792/0 conn(0x7fde0c0a3fe0 legacy=0x7fde0c0a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f780815a640 1 -- start start 2024-09-17T15:46:05.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f780815a640 10 monclient: init 2024-09-17T15:46:05.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 1 -- 172.21.15.64:0/1052846474 >> v1:172.21.15.64:6790/0 conn(0x7fde0c0a43c0 legacy=0x7fde0c0a47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 1 monclient: found mon.noname-c 2024-09-17T15:46:05.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:46:05.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 1 -- 172.21.15.64:0/1052846474 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0c0a1640 con 0x7fde0c13bfb0 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 1 -- 172.21.15.64:0/1052846474 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fde04001bb0 con 0x7fde0c13bfb0 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 10 monclient: dump: 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.902 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde12eed640 5 monclient: authenticate success, global_id 4378 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde12eed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 1 -- 172.21.15.64:0/1052846474 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fde04001d50 con 0x7fde0c13bfb0 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde12eed640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde12eed640 4 set_mon_vals no callback set 2024-09-17T15:46:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 1 -- 172.21.15.64:0/1052846474 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fde04001fc0 con 0x7fde0c13bfb0 2024-09-17T15:46:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7fde0b7fe640 10 monclient: dump: 2024-09-17T15:46:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.904 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.905 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.905 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.905 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.905 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.905 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.905 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.905 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.905 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f780815a640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f780815a640 10 monclient: _add_conns ranks=[2,1,3,8,7,4,5,0,6] 2024-09-17T15:46:05.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f780815a640 10 monclient(hunting): picked mon.noname-b con 0x7f78000688b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:05.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f780815a640 10 monclient(hunting): picked mon.noname-f con 0x7f78000a0c20 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:05.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f780815a640 10 monclient(hunting): picked mon.noname-g con 0x7f780013f620 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:05.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f780815a640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7800074bd0 con 0x7f78000a0c20 2024-09-17T15:46:05.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f780815a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7800142d30 con 0x7f78000688b0 2024-09-17T15:46:05.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f780815a640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78000722c0 con 0x7f780013f620 2024-09-17T15:46:05.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f780815a640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f780815a640 10 monclient(hunting): authenticate will time out at 763.479977s 2024-09-17T15:46:05.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f7805ef6640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f78000688b0 0x7f7800072930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46810/0 (socket says 172.21.15.64:46810) 2024-09-17T15:46:05.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.804+0000 7f7805ef6640 1 -- 172.21.15.64:0/1351467986 learned_addr learned my addr 172.21.15.64:0/1351467986 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.1 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f07a00647b0 con 0x7f07a01360a0 2024-09-17T15:46:05.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f0795ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f0795ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:05.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085332859 0 0) 0x7f7800142d30 con 0x7f78000688b0 2024-09-17T15:46:05.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): my global_id is 4331 2024-09-17T15:46:05.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77e4003520 con 0x7f78000688b0 2024-09-17T15:46:05.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7fde12eed640 10 monclient: shutdown 2024-09-17T15:46:05.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7fde12eed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7fde12eed640 1 -- 172.21.15.64:0/1052846474 >> v1:172.21.15.64:6791/0 conn(0x7fde0c13bfb0 legacy=0x7fde0c13e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f18fa589640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f18fa589640 10 monclient: monmap: 2024-09-17T15:46:05.909 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.909 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:05.909 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:05.909 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:05.909 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.909 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:05.909 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.909 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.910 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.910 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.910 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.910 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.910 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.910 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.910 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.911 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7fde12eed640 1 -- 172.21.15.64:0/1052846474 shutdown_connections 2024-09-17T15:46:05.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7fde12eed640 1 -- 172.21.15.64:0/1052846474 >> 172.21.15.64:0/1052846474 conn(0x7fde0c0a50d0 msgr2=0x7fde0c0a54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3075163658 0 0) 0x7f78000722c0 con 0x7f780013f620 2024-09-17T15:46:05.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): my global_id is 4368 2024-09-17T15:46:05.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7800142d30 con 0x7f780013f620 2024-09-17T15:46:05.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4029181204 0 0) 0x7f7800074bd0 con 0x7f78000a0c20 2024-09-17T15:46:05.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): my global_id is 4366 2024-09-17T15:46:05.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78000722c0 con 0x7f78000a0c20 2024-09-17T15:46:05.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2189770626 0 0) 0x7f77e4003520 con 0x7f78000688b0 2024-09-17T15:46:05.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7800074bd0 con 0x7f78000688b0 2024-09-17T15:46:05.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f77fc001a80 con 0x7f78000688b0 2024-09-17T15:46:05.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:05.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): dump: 2024-09-17T15:46:05.915 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.915 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.916 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.916 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.916 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.916 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.916 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.916 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.916 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.917 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.917 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.917 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.917 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.918 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.918 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.918 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.805+0000 7f77f67fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:05.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7fde12eed640 1 -- 172.21.15.64:0/1052846474 shutdown_connections 2024-09-17T15:46:05.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2482911187 0 0) 0x7f78000722c0 con 0x7f78000a0c20 2024-09-17T15:46:05.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77e4003520 con 0x7f78000a0c20 2024-09-17T15:46:05.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f77f0001a90 con 0x7f78000a0c20 2024-09-17T15:46:05.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:05.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 10 monclient(hunting): dump: 2024-09-17T15:46:05.920 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.920 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.920 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.921 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.921 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.921 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.921 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.921 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.921 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.921 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.921 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.921 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.921 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.921 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.922 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.922 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2119761597 0 0) 0x7f7800142d30 con 0x7f780013f620 2024-09-17T15:46:05.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78000722c0 con 0x7f780013f620 2024-09-17T15:46:05.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2077811702 0 0) 0x7f7800074bd0 con 0x7f78000688b0 2024-09-17T15:46:05.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 >> v1:172.21.15.100:6790/0 conn(0x7f780013f620 legacy=0x7f78001da7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 >> v1:172.21.15.100:6789/0 conn(0x7f78000a0c20 legacy=0x7f78001d7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 1 monclient: found mon.b 2024-09-17T15:46:05.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780009eb90 con 0x7f78000688b0 2024-09-17T15:46:05.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f77f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f780815a640 5 monclient: authenticate success, global_id 4331 2024-09-17T15:46:05.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f780815a640 10 monclient: _renew_subs 2024-09-17T15:46:05.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f780815a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f780815a640 1 -- 172.21.15.64:0/1351467986 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7800073ac0 con 0x7f78000688b0 2024-09-17T15:46:05.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f780815a640 10 monclient: _renew_subs 2024-09-17T15:46:05.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f780815a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f780815a640 1 -- 172.21.15.64:0/1351467986 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f780009ae80 con 0x7f78000688b0 2024-09-17T15:46:05.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f18fa589640 1 Processor -- start 2024-09-17T15:46:05.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f18fa589640 1 -- start start 2024-09-17T15:46:05.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7f18fa589640 10 monclient: init 2024-09-17T15:46:05.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.806+0000 7fde12eed640 1 -- 172.21.15.64:0/1052846474 wait complete. 2024-09-17T15:46:05.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f77fc0038c0 con 0x7f78000688b0 2024-09-17T15:46:05.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f77f67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7fde12eed640 10 monclient: build_initial_monmap 2024-09-17T15:46:05.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f18fa589640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f18fa589640 10 monclient: _add_conns ranks=[3,7,0,1,2,4,6,5,8] 2024-09-17T15:46:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7fde12eed640 10 monclient: monmap: 2024-09-17T15:46:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:05.927 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:05.927 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:05.927 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:05.927 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f77fc003e30 con 0x7f78000688b0 2024-09-17T15:46:05.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f18fa589640 10 monclient(hunting): picked mon.noname-g con 0x7f18f40689d0 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:05.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f77f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f77f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:05.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f18fa589640 10 monclient(hunting): picked mon.noname-i con 0x7f18f409e830 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:05.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f77f67fc640 10 monclient: dump: 2024-09-17T15:46:05.929 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.929 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.929 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.929 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.929 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.929 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.929 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.929 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.929 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.929 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.929 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.929 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.929 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.930 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.930 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.930 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f18f3fff640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f18f40689d0 0x7f18f41c9d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:47778/0 (socket says 172.21.15.64:47778) 2024-09-17T15:46:05.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f18f3fff640 1 -- 172.21.15.64:0/3583452905 learned_addr learned my addr 172.21.15.64:0/3583452905 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f77f7fff640 4 set_mon_vals no callback set 2024-09-17T15:46:05.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7fde12eed640 1 Processor -- start 2024-09-17T15:46:05.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7f18fa589640 10 monclient(hunting): picked mon.noname-a con 0x7f18f4137130 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:05.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7f18fa589640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18f4132910 con 0x7f18f4137130 2024-09-17T15:46:05.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7fde12eed640 1 -- start start 2024-09-17T15:46:05.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.807+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f77fc004090 con 0x7f78000688b0 2024-09-17T15:46:05.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f77fc04d8e0 con 0x7f78000688b0 2024-09-17T15:46:05.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7fde12eed640 10 monclient: init 2024-09-17T15:46:05.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7f78056f5640 10 monclient: get_auth_request con 0x7f77e4040c00 auth_method 0 2024-09-17T15:46:05.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7f780815a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2037585442 0 0) 0x7f18f4132910 con 0x7f18f4137130 2024-09-17T15:46:05.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7f780815a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7f780815a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:05.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7f18fa589640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18f4132bf0 con 0x7f18f40689d0 2024-09-17T15:46:05.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7f780815a640 1 -- 172.21.15.64:0/1351467986 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77e8005080 con 0x7f78000688b0 2024-09-17T15:46:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7f18fa589640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18f409dee0 con 0x7f18f409e830 2024-09-17T15:46:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7f18fa589640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.808+0000 7f18fa589640 10 monclient(hunting): authenticate will time out at 763.483921s 2024-09-17T15:46:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): my global_id is 4293 2024-09-17T15:46:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18d4003520 con 0x7f18f4137130 2024-09-17T15:46:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2437801944 0 0) 0x7f18f4132bf0 con 0x7f18f40689d0 2024-09-17T15:46:05.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7fde12eed640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:05.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7fde12eed640 10 monclient: _add_conns ranks=[4,5,7,1,0,8,3,6,2] 2024-09-17T15:46:05.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7fde12eed640 10 monclient(hunting): picked mon.noname-c con 0x7fde0c0a3fe0 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:05.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7fde12eed640 10 monclient(hunting): picked mon.noname-h con 0x7fde0c0a43c0 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:05.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde12eed640 10 monclient(hunting): picked mon.noname-i con 0x7fde0c13bfb0 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:05.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde12eed640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c12e740 con 0x7fde0c0a3fe0 2024-09-17T15:46:05.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde0bfff640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7fde0c0a43c0 0x7fde0c1ce820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:36770/0 (socket says 172.21.15.64:36770) 2024-09-17T15:46:05.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde10c89640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fde0c0a3fe0 0x7fde0c135cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:36814/0 (socket says 172.21.15.64:36814) 2024-09-17T15:46:05.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde0bfff640 1 -- 172.21.15.64:0/876660330 learned_addr learned my addr 172.21.15.64:0/876660330 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:05.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde12eed640 1 -- 172.21.15.64:0/876660330 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c0691c0 con 0x7fde0c0a43c0 2024-09-17T15:46:05.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde12eed640 1 -- 172.21.15.64:0/876660330 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c069560 con 0x7fde0c13bfb0 2024-09-17T15:46:05.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde12eed640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:05.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde12eed640 10 monclient(hunting): authenticate will time out at 763.485398s 2024-09-17T15:46:05.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): my global_id is 4377 2024-09-17T15:46:05.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18f4132910 con 0x7f18f40689d0 2024-09-17T15:46:05.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 331500444 0 0) 0x7f18f409dee0 con 0x7f18f409e830 2024-09-17T15:46:05.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): my global_id is 4345 2024-09-17T15:46:05.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18f4132bf0 con 0x7f18f409e830 2024-09-17T15:46:05.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 398026930 0 0) 0x7f18f4132bf0 con 0x7f18f409e830 2024-09-17T15:46:05.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18f409dee0 con 0x7f18f409e830 2024-09-17T15:46:05.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f18e4001ad0 con 0x7f18f409e830 2024-09-17T15:46:05.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-17T15:46:05.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): dump: 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.940 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.941 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.941 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.809+0000 7f18f17fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:05.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 270090918 0 0) 0x7f18d4003520 con 0x7f18f4137130 2024-09-17T15:46:05.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18f4132bf0 con 0x7f18f4137130 2024-09-17T15:46:05.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3120458228 0 0) 0x7f18f4132910 con 0x7f18f40689d0 2024-09-17T15:46:05.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18d4003520 con 0x7f18f40689d0 2024-09-17T15:46:05.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2917168640 0 0) 0x7f18f409dee0 con 0x7f18f409e830 2024-09-17T15:46:05.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 >> v1:172.21.15.100:6790/0 conn(0x7f18f40689d0 legacy=0x7f18f41c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 >> v1:172.21.15.64:6789/0 conn(0x7f18f4137130 legacy=0x7f18f41d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 1 monclient: found mon.i 2024-09-17T15:46:05.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:46:05.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.100:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f409d990 con 0x7f18f409e830 2024-09-17T15:46:05.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18fa589640 5 monclient: authenticate success, global_id 4345 2024-09-17T15:46:05.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18fa589640 10 monclient: _renew_subs 2024-09-17T15:46:05.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18fa589640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:46:05.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18fa589640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.100:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f18f40a0ff0 con 0x7f18f409e830 2024-09-17T15:46:05.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18fa589640 10 monclient: _renew_subs 2024-09-17T15:46:05.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18fa589640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:46:05.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18fa589640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.100:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f18f4132630 con 0x7f18f409e830 2024-09-17T15:46:05.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618986044 0 0) 0x7fde0c0691c0 con 0x7fde0c0a43c0 2024-09-17T15:46:05.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde097fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde097fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde097fa640 10 monclient(hunting): my global_id is 4316 2024-09-17T15:46:05.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddf0003520 con 0x7fde0c0a43c0 2024-09-17T15:46:05.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4002696502 0 0) 0x7fde0c12e740 con 0x7fde0c0a3fe0 2024-09-17T15:46:05.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde097fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde097fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.7 v1:172.21.15.100:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18e4001c70 con 0x7f18f409e830 2024-09-17T15:46:05.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.7 v1:172.21.15.100:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f18e4003be0 con 0x7f18f409e830 2024-09-17T15:46:05.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde097fa640 10 monclient(hunting): my global_id is 4387 2024-09-17T15:46:05.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde0c0691c0 con 0x7fde0c0a3fe0 2024-09-17T15:46:05.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3999204678 0 0) 0x7fddf0003520 con 0x7fde0c0a43c0 2024-09-17T15:46:05.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde0c12e740 con 0x7fde0c0a43c0 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fddf8001b90 con 0x7fde0c0a43c0 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 10 monclient(hunting): dump: 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.946 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.947 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.947 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.947 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.947 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.947 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.947 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:05.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:46:05.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f17fa640 10 monclient: dump: 2024-09-17T15:46:05.948 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.948 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.948 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.948 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.948 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.948 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.948 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.948 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.948 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.948 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.948 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.948 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.949 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.949 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.949 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.949 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.810+0000 7f18f2ffd640 4 set_mon_vals no callback set 2024-09-17T15:46:05.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.7 v1:172.21.15.100:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f18e40104c0 con 0x7f18f409e830 2024-09-17T15:46:05.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2085352395 0 0) 0x7fde0c069560 con 0x7fde0c13bfb0 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 10 monclient(hunting): my global_id is 4354 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddf0003520 con 0x7fde0c13bfb0 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 713107912 0 0) 0x7fde0c0691c0 con 0x7fde0c0a3fe0 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde0c069560 con 0x7fde0c0a3fe0 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fde00001430 con 0x7fde0c0a3fe0 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.811+0000 7fde097fa640 10 monclient(hunting): dump: 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.950 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3204962046 0 0) 0x7fde0c12e740 con 0x7fde0c0a43c0 2024-09-17T15:46:05.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.7 v1:172.21.15.100:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f18e404d720 con 0x7f18f409e830 2024-09-17T15:46:05.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 >> v1:172.21.15.100:6792/0 conn(0x7fde0c13bfb0 legacy=0x7fde0c1d1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7f18fa589640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7f18fa589640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7f18fa589640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:46:05.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7f18fa589640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.100:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18d8005080 con 0x7f18f409e830 2024-09-17T15:46:05.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7f18f3fff640 10 monclient: get_auth_request con 0x7f18d4048f30 auth_method 0 2024-09-17T15:46:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 >> v1:172.21.15.64:6791/0 conn(0x7fde0c0a3fe0 legacy=0x7fde0c135cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 10 monclient: _finish_hunting 0 2024-09-17T15:46:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 1 monclient: found mon.h 2024-09-17T15:46:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:46:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 --> v1:172.21.15.100:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0c12dea0 con 0x7fde0c0a43c0 2024-09-17T15:46:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 10 monclient: _finish_auth 0 2024-09-17T15:46:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 10 monclient: _check_auth_tickets 2024-09-17T15:46:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde12eed640 5 monclient: authenticate success, global_id 4316 2024-09-17T15:46:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.5 v1:172.21.15.100:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fddf80038a0 con 0x7fde0c0a43c0 2024-09-17T15:46:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.5 v1:172.21.15.100:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fddf8003e90 con 0x7fde0c0a43c0 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde097fa640 10 monclient: dump: 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:05.958 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.812+0000 7fde0a7fc640 4 set_mon_vals no callback set 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.813+0000 7fde12eed640 10 monclient: _renew_subs 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.813+0000 7fde12eed640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.813+0000 7fde12eed640 1 -- 172.21.15.64:0/876660330 --> v1:172.21.15.100:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fde0c12e180 con 0x7fde0c0a43c0 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.813+0000 7fde12eed640 10 monclient: _renew_subs 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.813+0000 7fde12eed640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.813+0000 7fde12eed640 1 -- 172.21.15.64:0/876660330 --> v1:172.21.15.100:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fde0c139e50 con 0x7fde0c0a43c0 2024-09-17T15:46:05.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.814+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.5 v1:172.21.15.100:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fddf80040b0 con 0x7fde0c0a43c0 2024-09-17T15:46:05.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.814+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.5 v1:172.21.15.100:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7fddf804eb70 con 0x7fde0c0a43c0 2024-09-17T15:46:05.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.815+0000 7fde10c89640 10 monclient: get_auth_request con 0x7fddf0040860 auth_method 0 2024-09-17T15:46:05.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.815+0000 7fde12eed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.815+0000 7fde12eed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.815+0000 7fde12eed640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:46:05.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.815+0000 7fde12eed640 1 -- 172.21.15.64:0/876660330 --> v1:172.21.15.100:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde0c073820 con 0x7fde0c0a43c0 2024-09-17T15:46:05.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.818+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.7 v1:172.21.15.100:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f18e4016340 con 0x7f18f409e830 2024-09-17T15:46:05.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.818+0000 7f18f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.818+0000 7f18f17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:05.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.818+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f77fc0176c0 con 0x7f78000688b0 2024-09-17T15:46:05.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.818+0000 7f77f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.818+0000 7f77f67fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:05.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.820+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.5 v1:172.21.15.100:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fddf801bb10 con 0x7fde0c0a43c0 2024-09-17T15:46:05.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.820+0000 7fde097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:05.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.820+0000 7fde097fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:05.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.991+0000 7fd2f3a33640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-17T15:46:05.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.991+0000 7fd2f3a33640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-17T15:46:05.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.991+0000 7fd2f3a33640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:46:05.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.991+0000 7fd2f3a33640 1 -- 172.21.15.64:0/3552820437 --> v1:172.21.15.100:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd2cc005370 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.991+0000 7fd2e1ffb640 1 -- 172.21.15.64:0/3552820437 <== mon.5 v1:172.21.15.100:6791/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 3756020955) 0x7fd2e4015f80 con 0x7fd2ec0a3ae0 2024-09-17T15:46:05.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.991+0000 7fd2e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-17T15:46:05.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.991+0000 7fd2e1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:05.994 INFO:teuthology.orchestra.run.smithi064.stdout:60129542150 2024-09-17T15:46:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.994+0000 7fd2f3a33640 1 -- 172.21.15.64:0/3552820437 >> v1:172.21.15.100:6812/4210963484 conn(0x7fd2d0040d90 legacy=0x7fd2d0043250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.994+0000 7fd2f3a33640 10 monclient: shutdown 2024-09-17T15:46:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.994+0000 7fd2f3a33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.994+0000 7fd2f3a33640 1 -- 172.21.15.64:0/3552820437 >> v1:172.21.15.100:6791/0 conn(0x7fd2ec0a3ae0 legacy=0x7fd2ec1d6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.994+0000 7fd2f3a33640 1 -- 172.21.15.64:0/3552820437 shutdown_connections 2024-09-17T15:46:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.994+0000 7fd2f3a33640 1 -- 172.21.15.64:0/3552820437 >> 172.21.15.64:0/3552820437 conn(0x7fd2ec0a87b0 msgr2=0x7fd2ec1c00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.994+0000 7fd2f3a33640 1 -- 172.21.15.64:0/3552820437 shutdown_connections 2024-09-17T15:46:05.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:05.994+0000 7fd2f3a33640 1 -- 172.21.15.64:0/3552820437 wait complete. 2024-09-17T15:46:06.007 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.1 2024-09-17T15:46:06.007 DEBUG:teuthology.parallel:result is None 2024-09-17T15:46:06.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.016+0000 7f18fa589640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T15:46:06.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.016+0000 7f18fa589640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T15:46:06.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.016+0000 7f18fa589640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:46:06.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.016+0000 7f18fa589640 1 -- 172.21.15.64:0/3583452905 --> v1:172.21.15.100:6792/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f18d8005370 con 0x7f18f409e830 2024-09-17T15:46:06.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.017+0000 7f18f17fa640 1 -- 172.21.15.64:0/3583452905 <== mon.7 v1:172.21.15.100:6792/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 3426929836) 0x7f18e4015c40 con 0x7f18f409e830 2024-09-17T15:46:06.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.017+0000 7f18f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T15:46:06.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.017+0000 7f18f17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:06.019 INFO:teuthology.orchestra.run.smithi064.stdout:60129542151 2024-09-17T15:46:06.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.019+0000 7f18fa589640 1 -- 172.21.15.64:0/3583452905 >> v1:172.21.15.100:6812/4210963484 conn(0x7f18d4048f30 legacy=0x7f18d404b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.019+0000 7f18fa589640 10 monclient: shutdown 2024-09-17T15:46:06.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.019+0000 7f18fa589640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:06.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.019+0000 7f18fa589640 1 -- 172.21.15.64:0/3583452905 >> v1:172.21.15.100:6792/0 conn(0x7f18f409e830 legacy=0x7f18f41cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.019+0000 7f18fa589640 1 -- 172.21.15.64:0/3583452905 shutdown_connections 2024-09-17T15:46:06.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.019+0000 7f18fa589640 1 -- 172.21.15.64:0/3583452905 >> 172.21.15.64:0/3583452905 conn(0x7f18f40a4e80 msgr2=0x7f18f40a42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:06.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.019+0000 7f18fa589640 1 -- 172.21.15.64:0/3583452905 shutdown_connections 2024-09-17T15:46:06.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.019+0000 7f18fa589640 1 -- 172.21.15.64:0/3583452905 wait complete. 2024-09-17T15:46:06.031 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.2 2024-09-17T15:46:06.031 DEBUG:teuthology.parallel:result is None 2024-09-17T15:46:06.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.106+0000 7f9ad6ffd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-17T15:46:06.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.106+0000 7f9ad6ffd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-17T15:46:06.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.106+0000 7f9ad6ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:06.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.106+0000 7f9ad6ffd640 1 -- 172.21.15.64:0/1978790322 --> v1:172.21.15.64:6792/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f9acc005370 con 0x7f9aec135c80 2024-09-17T15:46:06.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.106+0000 7f9ae97fa640 1 -- 172.21.15.64:0/1978790322 <== mon.6 v1:172.21.15.64:6792/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 1169801742) 0x7f9ad8017630 con 0x7f9aec135c80 2024-09-17T15:46:06.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.106+0000 7f9ae97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-17T15:46:06.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.106+0000 7f9ae97fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:06.109 INFO:teuthology.orchestra.run.smithi064.stdout:47244640261 2024-09-17T15:46:06.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.109+0000 7f9ad6ffd640 1 -- 172.21.15.64:0/1978790322 >> v1:172.21.15.100:6812/4210963484 conn(0x7f9ad0040470 legacy=0x7f9ad0042930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.109+0000 7f9ad6ffd640 10 monclient: shutdown 2024-09-17T15:46:06.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.109+0000 7f9ad6ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:06.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.109+0000 7f9ad6ffd640 1 -- 172.21.15.64:0/1978790322 >> v1:172.21.15.64:6792/0 conn(0x7f9aec135c80 legacy=0x7f9aec1d71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.109+0000 7f9ad6ffd640 1 -- 172.21.15.64:0/1978790322 shutdown_connections 2024-09-17T15:46:06.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.109+0000 7f9ad6ffd640 1 -- 172.21.15.64:0/1978790322 >> 172.21.15.64:0/1978790322 conn(0x7f9aec06daa0 msgr2=0x7f9aec0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:06.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.109+0000 7f9ad6ffd640 1 -- 172.21.15.64:0/1978790322 shutdown_connections 2024-09-17T15:46:06.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.109+0000 7f9ad6ffd640 1 -- 172.21.15.64:0/1978790322 wait complete. 2024-09-17T15:46:06.121 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.5 2024-09-17T15:46:06.122 DEBUG:teuthology.parallel:result is None 2024-09-17T15:46:06.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.128+0000 7f07a74a3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-17T15:46:06.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.128+0000 7f07a74a3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-17T15:46:06.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.128+0000 7f07a74a3640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:46:06.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.128+0000 7f07a74a3640 1 -- 172.21.15.64:0/3358980176 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f07a00a5220 con 0x7f07a01360a0 2024-09-17T15:46:06.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.128+0000 7f0795ffb640 1 -- 172.21.15.64:0/3358980176 <== mon.1 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 1902013079) 0x7f0798016740 con 0x7f07a01360a0 2024-09-17T15:46:06.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.128+0000 7f0795ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-17T15:46:06.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.128+0000 7f0795ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:06.130 INFO:teuthology.orchestra.run.smithi064.stdout:47244640262 2024-09-17T15:46:06.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.130+0000 7f07a74a3640 1 -- 172.21.15.64:0/3358980176 >> v1:172.21.15.100:6812/4210963484 conn(0x7f0784041010 legacy=0x7f07840434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.130+0000 7f07a74a3640 10 monclient: shutdown 2024-09-17T15:46:06.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.130+0000 7f07a74a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:06.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.130+0000 7f07a74a3640 1 -- 172.21.15.64:0/3358980176 >> v1:172.21.15.100:6789/0 conn(0x7f07a01360a0 legacy=0x7f07a01d5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.130+0000 7f780815a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-17T15:46:06.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.130+0000 7f780815a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-17T15:46:06.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.130+0000 7f780815a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:06.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.130+0000 7f07a74a3640 1 -- 172.21.15.64:0/3358980176 shutdown_connections 2024-09-17T15:46:06.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.130+0000 7f07a74a3640 1 -- 172.21.15.64:0/3358980176 >> 172.21.15.64:0/3358980176 conn(0x7f07a006daa0 msgr2=0x7f07a00a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:06.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.130+0000 7f780815a640 1 -- 172.21.15.64:0/1351467986 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f77e8005370 con 0x7f78000688b0 2024-09-17T15:46:06.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.130+0000 7f07a74a3640 1 -- 172.21.15.64:0/3358980176 shutdown_connections 2024-09-17T15:46:06.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.131+0000 7f07a74a3640 1 -- 172.21.15.64:0/3358980176 wait complete. 2024-09-17T15:46:06.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.131+0000 7f77f67fc640 1 -- 172.21.15.64:0/1351467986 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 4171588661) 0x7f77fc016fc0 con 0x7f78000688b0 2024-09-17T15:46:06.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.131+0000 7f77f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-17T15:46:06.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.131+0000 7f77f67fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:06.134 INFO:teuthology.orchestra.run.smithi064.stdout:60129542152 2024-09-17T15:46:06.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.133+0000 7f780815a640 1 -- 172.21.15.64:0/1351467986 >> v1:172.21.15.100:6812/4210963484 conn(0x7f77e4040c00 legacy=0x7f77e40430c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.133+0000 7f780815a640 10 monclient: shutdown 2024-09-17T15:46:06.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.133+0000 7f780815a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:06.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.133+0000 7f780815a640 1 -- 172.21.15.64:0/1351467986 >> v1:172.21.15.64:6790/0 conn(0x7f78000688b0 legacy=0x7f7800072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.133+0000 7f780815a640 1 -- 172.21.15.64:0/1351467986 shutdown_connections 2024-09-17T15:46:06.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.133+0000 7f780815a640 1 -- 172.21.15.64:0/1351467986 >> 172.21.15.64:0/1351467986 conn(0x7f78000a8640 msgr2=0x7f78000a4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:06.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.133+0000 7f780815a640 1 -- 172.21.15.64:0/1351467986 shutdown_connections 2024-09-17T15:46:06.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.133+0000 7f780815a640 1 -- 172.21.15.64:0/1351467986 wait complete. 2024-09-17T15:46:06.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.138+0000 7fde12eed640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-17T15:46:06.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.138+0000 7fde12eed640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-17T15:46:06.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.138+0000 7fde12eed640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:46:06.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.138+0000 7fde12eed640 1 -- 172.21.15.64:0/876660330 --> v1:172.21.15.100:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fde0c06e8e0 con 0x7fde0c0a43c0 2024-09-17T15:46:06.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.139+0000 7fde097fa640 1 -- 172.21.15.64:0/876660330 <== mon.5 v1:172.21.15.100:6791/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 3756282479) 0x7fddf8020a50 con 0x7fde0c0a43c0 2024-09-17T15:46:06.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.139+0000 7fde097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-17T15:46:06.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.139+0000 7fde097fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:06.141 INFO:teuthology.orchestra.run.smithi064.stdout:55834574854 2024-09-17T15:46:06.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.141+0000 7fde12eed640 1 -- 172.21.15.64:0/876660330 >> v1:172.21.15.100:6812/4210963484 conn(0x7fddf0040860 legacy=0x7fddf0042d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.141+0000 7fde12eed640 10 monclient: shutdown 2024-09-17T15:46:06.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.141+0000 7fde12eed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:06.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.141+0000 7fde12eed640 1 -- 172.21.15.64:0/876660330 >> v1:172.21.15.100:6791/0 conn(0x7fde0c0a43c0 legacy=0x7fde0c1ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.141+0000 7fde12eed640 1 -- 172.21.15.64:0/876660330 shutdown_connections 2024-09-17T15:46:06.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.141+0000 7fde12eed640 1 -- 172.21.15.64:0/876660330 >> 172.21.15.64:0/876660330 conn(0x7fde0c0a50d0 msgr2=0x7fde0c13b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:06.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.141+0000 7fde12eed640 1 -- 172.21.15.64:0/876660330 shutdown_connections 2024-09-17T15:46:06.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.141+0000 7fde12eed640 1 -- 172.21.15.64:0/876660330 wait complete. 2024-09-17T15:46:06.145 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.3 2024-09-17T15:46:06.145 DEBUG:teuthology.parallel:result is None 2024-09-17T15:46:06.146 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.0 2024-09-17T15:46:06.146 DEBUG:teuthology.parallel:result is None 2024-09-17T15:46:06.153 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2024-09-17T15:46:06.153 DEBUG:teuthology.parallel:result is None 2024-09-17T15:46:06.153 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T15:46:06.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:46:06.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.293+0000 7f3bcbd10640 10 monclient: get_monmap_and_config 2024-09-17T15:46:06.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.293+0000 7f3bcbd10640 10 monclient: build_initial_monmap 2024-09-17T15:46:06.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.293+0000 7f3bcbd10640 10 monclient: monmap: 2024-09-17T15:46:06.295 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:06.295 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.295 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:06.294075+0000 2024-09-17T15:46:06.295 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:06.294075+0000 2024-09-17T15:46:06.295 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:06.295 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.295 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:06.295 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:06.295 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:06.295 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:06.295 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:06.296 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:06.296 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:06.296 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:06.296 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:06.296 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.293+0000 7f3bcbd10640 1 Processor -- start 2024-09-17T15:46:06.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bcbd10640 1 -- start start 2024-09-17T15:46:06.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bcbd10640 10 monclient: init 2024-09-17T15:46:06.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bcbd10640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:06.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bcbd10640 10 monclient: _add_conns ranks=[8,6,1,5,3,4,2,0,7] 2024-09-17T15:46:06.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bcbd10640 10 monclient(hunting): picked mon.noname-e con 0x7f3bc40a54b0 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:06.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bcbd10640 10 monclient(hunting): picked mon.noname-d con 0x7f3bc4135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:06.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bcbd10640 10 monclient(hunting): picked mon.noname-f con 0x7f3bc41360a0 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:06.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bcbd10640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc40aacb0 con 0x7f3bc41360a0 2024-09-17T15:46:06.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bcbd10640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc409d780 con 0x7f3bc4135c80 2024-09-17T15:46:06.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bcbd10640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc409da60 con 0x7f3bc40a54b0 2024-09-17T15:46:06.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bcbd10640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:06.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bcbd10640 10 monclient(hunting): authenticate will time out at 763.969737s 2024-09-17T15:46:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bc9aac640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f3bc40a54b0 0x7f3bc40a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38644/0 (socket says 172.21.15.64:38644) 2024-09-17T15:46:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bc92ab640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f3bc4135c80 0x7f3bc413df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:49496/0 (socket says 172.21.15.64:49496) 2024-09-17T15:46:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.294+0000 7f3bc9aac640 1 -- 172.21.15.64:0/804628081 learned_addr learned my addr 172.21.15.64:0/804628081 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 -- 172.21.15.64:0/804628081 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3551974750 0 0) 0x7f3bc409da60 con 0x7f3bc40a54b0 2024-09-17T15:46:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient(hunting): my global_id is 4346 2024-09-17T15:46:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 -- 172.21.15.64:0/804628081 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bb0003520 con 0x7f3bc40a54b0 2024-09-17T15:46:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 -- 172.21.15.64:0/804628081 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2783921553 0 0) 0x7f3bc409d780 con 0x7f3bc4135c80 2024-09-17T15:46:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient(hunting): my global_id is 4308 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 -- 172.21.15.64:0/804628081 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bc409da60 con 0x7f3bc4135c80 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 -- 172.21.15.64:0/804628081 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1622674436 0 0) 0x7f3bc40aacb0 con 0x7f3bc41360a0 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient(hunting): my global_id is 4375 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 -- 172.21.15.64:0/804628081 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bc409d780 con 0x7f3bc41360a0 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 -- 172.21.15.64:0/804628081 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 578817900 0 0) 0x7f3bc409da60 con 0x7f3bc4135c80 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 -- 172.21.15.64:0/804628081 >> v1:172.21.15.64:6793/0 conn(0x7f3bc40a54b0 legacy=0x7f3bc40a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 -- 172.21.15.64:0/804628081 >> v1:172.21.15.100:6789/0 conn(0x7f3bc41360a0 legacy=0x7f3bc4141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient: _finish_hunting 0 2024-09-17T15:46:06.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 monclient: found mon.noname-d 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 -- 172.21.15.64:0/804628081 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc409dd40 con 0x7f3bc4135c80 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient: _finish_auth 0 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient: _check_auth_tickets 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 -- 172.21.15.64:0/804628081 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3bc0001810 con 0x7f3bc4135c80 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bcbd10640 5 monclient: authenticate success, global_id 4308 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bcbd10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient: dump: 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.307 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.307 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.307 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.307 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.307 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.307 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.307 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.307 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.307 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.307 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.307 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.307 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bcbd10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 -- 172.21.15.64:0/804628081 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3bc00019b0 con 0x7f3bc4135c80 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bcbd10640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 1 -- 172.21.15.64:0/804628081 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3bc0004b20 con 0x7f3bc4135c80 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bcbd10640 4 set_mon_vals no callback set 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bc8aaa640 10 monclient: dump: 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.308 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.309 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.309 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.309 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.309 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.309 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.309 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.309 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.309 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.309 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bcbd10640 10 monclient: shutdown 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bcbd10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.295+0000 7f3bcbd10640 1 -- 172.21.15.64:0/804628081 >> v1:172.21.15.64:6792/0 conn(0x7f3bc4135c80 legacy=0x7f3bc413df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.296+0000 7f3bcbd10640 1 -- 172.21.15.64:0/804628081 shutdown_connections 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.296+0000 7f3bcbd10640 1 -- 172.21.15.64:0/804628081 >> 172.21.15.64:0/804628081 conn(0x7f3bc406daa0 msgr2=0x7f3bc40ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.296+0000 7f3bcbd10640 1 -- 172.21.15.64:0/804628081 shutdown_connections 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.296+0000 7f3bcbd10640 1 -- 172.21.15.64:0/804628081 wait complete. 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.296+0000 7f3bcbd10640 10 monclient: build_initial_monmap 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.296+0000 7f3bcbd10640 10 monclient: monmap: 2024-09-17T15:46:06.310 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:06.311 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bcbd10640 1 Processor -- start 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bcbd10640 1 -- start start 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bcbd10640 10 monclient: init 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bcbd10640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bcbd10640 10 monclient: _add_conns ranks=[1,4,0,7,5,2,3,6,8] 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bcbd10640 10 monclient(hunting): picked mon.noname-f con 0x7f3bc40a54b0 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bcbd10640 10 monclient(hunting): picked mon.noname-c con 0x7f3bc4135c80 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bcbd10640 10 monclient(hunting): picked mon.noname-a con 0x7f3bc41360a0 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bcbd10640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc409f0a0 con 0x7f3bc41360a0 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bcbd10640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc40a6ec0 con 0x7f3bc40a54b0 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bcbd10640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc40a6900 con 0x7f3bc4135c80 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bcbd10640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:06.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bcbd10640 10 monclient(hunting): authenticate will time out at 763.972624s 2024-09-17T15:46:06.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bc92ab640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f3bc4135c80 0x7f3bc4132210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:36820/0 (socket says 172.21.15.64:36820) 2024-09-17T15:46:06.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bca2ad640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f3bc41360a0 0x7f3bc4132920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:58972/0 (socket says 172.21.15.64:58972) 2024-09-17T15:46:06.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bc92ab640 1 -- 172.21.15.64:0/1364878273 learned_addr learned my addr 172.21.15.64:0/1364878273 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:06.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1142029057 0 0) 0x7f3bc409dd40 con 0x7f3bc4135c80 2024-09-17T15:46:06.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 10 monclient(hunting): my global_id is 4396 2024-09-17T15:46:06.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ba8003520 con 0x7f3bc4135c80 2024-09-17T15:46:06.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 853611978 0 0) 0x7f3bc409d780 con 0x7f3bc41360a0 2024-09-17T15:46:06.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 10 monclient(hunting): my global_id is 4302 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bc409dd40 con 0x7f3bc41360a0 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4223651133 0 0) 0x7f3bb8001380 con 0x7f3bc40a54b0 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 10 monclient(hunting): my global_id is 4384 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.297+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bc409d780 con 0x7f3bc40a54b0 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2727002791 0 0) 0x7f3ba8003520 con 0x7f3bc4135c80 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3bb8001380 con 0x7f3bc4135c80 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 282310195 0 0) 0x7f3bc409d780 con 0x7f3bc40a54b0 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ba8003520 con 0x7f3bc40a54b0 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3bc00016d0 con 0x7f3bc4135c80 2024-09-17T15:46:06.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient(hunting): dump: 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.315 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.316 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 877000337 0 0) 0x7f3bc409dd40 con 0x7f3bc41360a0 2024-09-17T15:46:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3bc409d780 con 0x7f3bc41360a0 2024-09-17T15:46:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3bb8001950 con 0x7f3bc40a54b0 2024-09-17T15:46:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient(hunting): dump: 2024-09-17T15:46:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.316 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.317 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.318 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3bbc001970 con 0x7f3bc41360a0 2024-09-17T15:46:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient(hunting): dump: 2024-09-17T15:46:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.318 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3245427981 0 0) 0x7f3bb8001380 con 0x7f3bc4135c80 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 >> v1:172.21.15.100:6789/0 conn(0x7f3bc40a54b0 legacy=0x7f3bc4131b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 >> v1:172.21.15.64:6789/0 conn(0x7f3bc41360a0 legacy=0x7f3bc4132920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:06.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 monclient: found mon.c 2024-09-17T15:46:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc40a0520 con 0x7f3bc4135c80 2024-09-17T15:46:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bcbd10640 5 monclient: authenticate success, global_id 4396 2024-09-17T15:46:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bcbd10640 10 monclient: _renew_subs 2024-09-17T15:46:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bcbd10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bcbd10640 1 -- 172.21.15.64:0/1364878273 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc409e4a0 con 0x7f3bc4135c80 2024-09-17T15:46:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bcbd10640 10 monclient: _renew_subs 2024-09-17T15:46:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bcbd10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:06.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bcbd10640 1 -- 172.21.15.64:0/1364878273 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f3bc409f2d0 con 0x7f3bc4135c80 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3bc0004de0 con 0x7f3bc4135c80 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3bc00019c0 con 0x7f3bc4135c80 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb7fff640 4 set_mon_vals no callback set 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.298+0000 7f3bb67fc640 10 monclient: dump: 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.321 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.299+0000 7f3bcbd10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.299+0000 7f3bcbd10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.299+0000 7f3bcbd10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.299+0000 7f3bcbd10640 1 -- 172.21.15.64:0/1364878273 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bac005080 con 0x7f3bc4135c80 2024-09-17T15:46:06.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.299+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f3bc0011a10 con 0x7f3bc4135c80 2024-09-17T15:46:06.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.299+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f3bc004ea60 con 0x7f3bc4135c80 2024-09-17T15:46:06.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.300+0000 7f3bc9aac640 10 monclient: get_auth_request con 0x7f3ba8040cf0 auth_method 0 2024-09-17T15:46:06.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.304+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f3bc40a6900 con 0x7f3bc4135c80 2024-09-17T15:46:06.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.304+0000 7f3bb67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:06.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.304+0000 7f3bb67fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:06.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.488+0000 7f3bcbd10640 1 -- 172.21.15.64:0/1364878273 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3bac000b90 con 0x7f3ba8040cf0 2024-09-17T15:46:06.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.489+0000 7f3bb67fc640 1 -- 172.21.15.64:0/1364878273 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34213 (unknown 2965378022 0 3364237253) 0x7f3bac000b90 con 0x7f3ba8040cf0 2024-09-17T15:46:06.492 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:46:06.492 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-17T15:46:06.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.491+0000 7f3bcbd10640 1 -- 172.21.15.64:0/1364878273 >> v1:172.21.15.100:6812/4210963484 conn(0x7f3ba8040cf0 legacy=0x7f3ba80431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.491+0000 7f3bcbd10640 10 monclient: shutdown 2024-09-17T15:46:06.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.491+0000 7f3bcbd10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:06.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.491+0000 7f3bcbd10640 1 -- 172.21.15.64:0/1364878273 >> v1:172.21.15.64:6791/0 conn(0x7f3bc4135c80 legacy=0x7f3bc4132210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.491+0000 7f3bcbd10640 1 -- 172.21.15.64:0/1364878273 shutdown_connections 2024-09-17T15:46:06.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.491+0000 7f3bcbd10640 1 -- 172.21.15.64:0/1364878273 >> 172.21.15.64:0/1364878273 conn(0x7f3bc406daa0 msgr2=0x7f3bc40a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:06.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.492+0000 7f3bcbd10640 1 -- 172.21.15.64:0/1364878273 shutdown_connections 2024-09-17T15:46:06.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.492+0000 7f3bcbd10640 1 -- 172.21.15.64:0/1364878273 wait complete. 2024-09-17T15:46:06.503 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-17T15:46:05.833502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163244,"kb_used_data":1052,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067796,"statfs":{"total":579820584960,"available":579653423104,"internally_reserved":0,"allocated":1077248,"data_stored":1360066,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.906091"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952807+0000","last_change":"2024-09-17T15:46:03.953098+0000","last_active":"2024-09-17T15:46:03.952807+0000","last_peered":"2024-09-17T15:46:03.952807+0000","last_clean":"2024-09-17T15:46:03.952807+0000","last_became_active":"2024-09-17T15:46:02.372737+0000","last_became_peered":"2024-09-17T15:46:02.372737+0000","last_unstale":"2024-09-17T15:46:03.952807+0000","last_undegraded":"2024-09-17T15:46:03.952807+0000","last_fullsized":"2024-09-17T15:46:03.952807+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:04:26.679936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057658000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.954285+0000","last_change":"2024-09-17T15:46:03.954609+0000","last_active":"2024-09-17T15:46:03.954285+0000","last_peered":"2024-09-17T15:46:03.954285+0000","last_clean":"2024-09-17T15:46:03.954285+0000","last_became_active":"2024-09-17T15:46:01.943967+0000","last_became_peered":"2024-09-17T15:46:01.943967+0000","last_unstale":"2024-09-17T15:46:03.954285+0000","last_undegraded":"2024-09-17T15:46:03.954285+0000","last_fullsized":"2024-09-17T15:46:03.954285+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:44:26.433116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058926900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952760+0000","last_change":"2024-09-17T15:46:03.953301+0000","last_active":"2024-09-17T15:46:03.952760+0000","last_peered":"2024-09-17T15:46:03.952760+0000","last_clean":"2024-09-17T15:46:03.952760+0000","last_became_active":"2024-09-17T15:46:01.937233+0000","last_became_peered":"2024-09-17T15:46:01.937233+0000","last_unstale":"2024-09-17T15:46:03.952760+0000","last_undegraded":"2024-09-17T15:46:03.952760+0000","last_fullsized":"2024-09-17T15:46:03.952760+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:36:15.542838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001175981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.954344+0000","last_change":"2024-09-17T15:46:03.954824+0000","last_active":"2024-09-17T15:46:03.954344+0000","last_peered":"2024-09-17T15:46:03.954344+0000","last_clean":"2024-09-17T15:46:03.954344+0000","last_became_active":"2024-09-17T15:46:01.941419+0000","last_became_peered":"2024-09-17T15:46:01.941419+0000","last_unstale":"2024-09-17T15:46:03.954344+0000","last_undegraded":"2024-09-17T15:46:03.954344+0000","last_fullsized":"2024-09-17T15:46:03.954344+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:53:39.894895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077909099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952714+0000","last_change":"2024-09-17T15:46:03.952714+0000","last_active":"2024-09-17T15:46:03.952714+0000","last_peered":"2024-09-17T15:46:03.952714+0000","last_clean":"2024-09-17T15:46:03.952714+0000","last_became_active":"2024-09-17T15:46:01.941164+0000","last_became_peered":"2024-09-17T15:46:01.941164+0000","last_unstale":"2024-09-17T15:46:03.952714+0000","last_undegraded":"2024-09-17T15:46:03.952714+0000","last_fullsized":"2024-09-17T15:46:03.952714+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:07:59.957722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051738399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.954670+0000","last_change":"2024-09-17T15:46:03.954913+0000","last_active":"2024-09-17T15:46:03.954670+0000","last_peered":"2024-09-17T15:46:03.954670+0000","last_clean":"2024-09-17T15:46:03.954670+0000","last_became_active":"2024-09-17T15:46:01.941188+0000","last_became_peered":"2024-09-17T15:46:01.941188+0000","last_unstale":"2024-09-17T15:46:03.954670+0000","last_undegraded":"2024-09-17T15:46:03.954670+0000","last_fullsized":"2024-09-17T15:46:03.954670+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:25:32.291552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087819899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.953563+0000","last_change":"2024-09-17T15:46:03.953722+0000","last_active":"2024-09-17T15:46:03.953563+0000","last_peered":"2024-09-17T15:46:03.953563+0000","last_clean":"2024-09-17T15:46:03.953563+0000","last_became_active":"2024-09-17T15:46:01.941223+0000","last_became_peered":"2024-09-17T15:46:01.941223+0000","last_unstale":"2024-09-17T15:46:03.953563+0000","last_undegraded":"2024-09-17T15:46:03.953563+0000","last_fullsized":"2024-09-17T15:46:03.953563+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:18:15.696629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012017600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952412+0000","last_change":"2024-09-17T15:46:03.952788+0000","last_active":"2024-09-17T15:46:03.952412+0000","last_peered":"2024-09-17T15:46:03.952412+0000","last_clean":"2024-09-17T15:46:03.952412+0000","last_became_active":"2024-09-17T15:46:01.937420+0000","last_became_peered":"2024-09-17T15:46:01.937420+0000","last_unstale":"2024-09-17T15:46:03.952412+0000","last_undegraded":"2024-09-17T15:46:03.952412+0000","last_fullsized":"2024-09-17T15:46:03.952412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:03:18.015826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065159300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'72","reported_seq":121,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952920+0000","last_change":"2024-09-17T15:45:56.919389+0000","last_active":"2024-09-17T15:46:03.952920+0000","last_peered":"2024-09-17T15:46:03.952920+0000","last_clean":"2024-09-17T15:46:03.952920+0000","last_became_active":"2024-09-17T15:45:56.908629+0000","last_became_peered":"2024-09-17T15:45:56.908629+0000","last_unstale":"2024-09-17T15:46:03.952920+0000","last_undegraded":"2024-09-17T15:46:03.952920+0000","last_fullsized":"2024-09-17T15:46:03.952920+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_clean_scrub_stamp":"2024-09-17T15:45:53.875287+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:44:47.696204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":1057856,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":51050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":51069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":196608,"data_stored":575850,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":51050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":51069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":579978,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:46:06.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:46:06.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.604+0000 7ff34acbd640 10 monclient: get_monmap_and_config 2024-09-17T15:46:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.604+0000 7ff34acbd640 10 monclient: build_initial_monmap 2024-09-17T15:46:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.604+0000 7ff34acbd640 10 monclient: monmap: 2024-09-17T15:46:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:06.605716+0000 2024-09-17T15:46:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:06.605716+0000 2024-09-17T15:46:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:06.608 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:06.609 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:06.609 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:06.609 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:06.609 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:06.609 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.605+0000 7ff34acbd640 1 Processor -- start 2024-09-17T15:46:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.605+0000 7ff34acbd640 1 -- start start 2024-09-17T15:46:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.605+0000 7ff34acbd640 10 monclient: init 2024-09-17T15:46:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff34acbd640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:06.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff34acbd640 10 monclient: _add_conns ranks=[7,4,0,3,2,6,5,8,1] 2024-09-17T15:46:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff34acbd640 10 monclient(hunting): picked mon.noname-i con 0x7ff3440a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff34acbd640 10 monclient(hunting): picked mon.noname-c con 0x7ff344135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff34acbd640 10 monclient(hunting): picked mon.noname-a con 0x7ff344135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff34acbd640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff34409d6f0 con 0x7ff344135f70 2024-09-17T15:46:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff34acbd640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff34409d9d0 con 0x7ff344135b50 2024-09-17T15:46:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff34acbd640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff34409dcb0 con 0x7ff3440a5520 2024-09-17T15:46:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff34acbd640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff34acbd640 10 monclient(hunting): authenticate will time out at 764.281265s 2024-09-17T15:46:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff343fff640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7ff344135b50 0x7ff34413deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:36826/0 (socket says 172.21.15.64:36826) 2024-09-17T15:46:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff343fff640 1 -- 172.21.15.64:0/492951681 learned_addr learned my addr 172.21.15.64:0/492951681 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 1 -- 172.21.15.64:0/492951681 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 475532327 0 0) 0x7ff34409d6f0 con 0x7ff344135f70 2024-09-17T15:46:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient(hunting): my global_id is 4311 2024-09-17T15:46:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 1 -- 172.21.15.64:0/492951681 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff330003520 con 0x7ff344135f70 2024-09-17T15:46:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 1 -- 172.21.15.64:0/492951681 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1757282084 0 0) 0x7ff34409d9d0 con 0x7ff344135b50 2024-09-17T15:46:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient(hunting): my global_id is 4405 2024-09-17T15:46:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 1 -- 172.21.15.64:0/492951681 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff34409d6f0 con 0x7ff344135b50 2024-09-17T15:46:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 1 -- 172.21.15.64:0/492951681 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355446065 0 0) 0x7ff34409dcb0 con 0x7ff3440a5520 2024-09-17T15:46:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient(hunting): my global_id is 4363 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 1 -- 172.21.15.64:0/492951681 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff34409d9d0 con 0x7ff3440a5520 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 1 -- 172.21.15.64:0/492951681 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 471585747 0 0) 0x7ff330003520 con 0x7ff344135f70 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 1 -- 172.21.15.64:0/492951681 >> v1:172.21.15.100:6792/0 conn(0x7ff3440a5520 legacy=0x7ff3440a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 1 -- 172.21.15.64:0/492951681 >> v1:172.21.15.64:6791/0 conn(0x7ff344135b50 legacy=0x7ff34413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 1 monclient: found mon.noname-a 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 1 -- 172.21.15.64:0/492951681 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff34409ccd0 con 0x7ff344135f70 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:06.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 1 -- 172.21.15.64:0/492951681 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff33c001830 con 0x7ff344135f70 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff34acbd640 5 monclient: authenticate success, global_id 4311 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff34acbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff3437fe640 10 monclient: dump: 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.622 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.606+0000 7ff34acbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff3437fe640 1 -- 172.21.15.64:0/492951681 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff33c0019d0 con 0x7ff344135f70 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff3437fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff3437fe640 1 -- 172.21.15.64:0/492951681 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff33c003950 con 0x7ff344135f70 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff34acbd640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff34acbd640 4 set_mon_vals no callback set 2024-09-17T15:46:06.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff3437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff3437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff3437fe640 10 monclient: dump: 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.624 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.625 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.625 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.625 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.625 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff34acbd640 10 monclient: shutdown 2024-09-17T15:46:06.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff34acbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:06.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff34acbd640 1 -- 172.21.15.64:0/492951681 >> v1:172.21.15.64:6789/0 conn(0x7ff344135f70 legacy=0x7ff3441415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff34acbd640 1 -- 172.21.15.64:0/492951681 shutdown_connections 2024-09-17T15:46:06.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff34acbd640 1 -- 172.21.15.64:0/492951681 >> 172.21.15.64:0/492951681 conn(0x7ff34406db30 msgr2=0x7ff3440ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:06.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff34acbd640 1 -- 172.21.15.64:0/492951681 shutdown_connections 2024-09-17T15:46:06.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.607+0000 7ff34acbd640 1 -- 172.21.15.64:0/492951681 wait complete. 2024-09-17T15:46:06.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.608+0000 7ff34acbd640 10 monclient: build_initial_monmap 2024-09-17T15:46:06.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.608+0000 7ff34acbd640 10 monclient: monmap: 2024-09-17T15:46:06.626 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:06.626 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:06.626 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:06.626 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:06.626 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:06.626 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:06.626 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:06.626 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:06.626 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:06.626 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:06.626 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:06.626 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:06.626 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.608+0000 7ff34acbd640 1 Processor -- start 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.608+0000 7ff34acbd640 1 -- start start 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.608+0000 7ff34acbd640 10 monclient: init 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff34acbd640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff34acbd640 10 monclient: _add_conns ranks=[2,3,4,6,0,5,1,7,8] 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff34acbd640 10 monclient(hunting): picked mon.noname-b con 0x7ff3440a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff34acbd640 10 monclient(hunting): picked mon.noname-g con 0x7ff344135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff34acbd640 10 monclient(hunting): picked mon.noname-c con 0x7ff344135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff34acbd640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff34413a280 con 0x7ff3440a5520 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff34acbd640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff344135310 con 0x7ff344135b50 2024-09-17T15:46:06.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff34acbd640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff344134f50 con 0x7ff344135f70 2024-09-17T15:46:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff34acbd640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff34acbd640 10 monclient(hunting): authenticate will time out at 764.284145s 2024-09-17T15:46:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff348a59640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7ff3440a5520 0x7ff344136dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46814/0 (socket says 172.21.15.64:46814) 2024-09-17T15:46:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff348a59640 1 -- 172.21.15.64:0/2010866184 learned_addr learned my addr 172.21.15.64:0/2010866184 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 588406468 0 0) 0x7ff34409d9d0 con 0x7ff3440a5520 2024-09-17T15:46:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): my global_id is 4340 2024-09-17T15:46:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff324003550 con 0x7ff3440a5520 2024-09-17T15:46:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3292757335 0 0) 0x7ff344134f50 con 0x7ff344135f70 2024-09-17T15:46:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): my global_id is 4414 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff34409d9d0 con 0x7ff344135f70 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771407455 0 0) 0x7ff344135310 con 0x7ff344135b50 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): my global_id is 4386 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff344134f50 con 0x7ff344135b50 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 697099065 0 0) 0x7ff324003550 con 0x7ff3440a5520 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff344135310 con 0x7ff3440a5520 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff32c001bb0 con 0x7ff3440a5520 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:06.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): dump: 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.630 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1994455390 0 0) 0x7ff34409d9d0 con 0x7ff344135f70 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff324003550 con 0x7ff344135f70 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff33c001770 con 0x7ff344135f70 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): dump: 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.631 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 409480617 0 0) 0x7ff344134f50 con 0x7ff344135b50 2024-09-17T15:46:06.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff34409d9d0 con 0x7ff344135b50 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff3340018d0 con 0x7ff344135b50 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 10 monclient(hunting): dump: 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.633 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.634 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4237065931 0 0) 0x7ff344135310 con 0x7ff3440a5520 2024-09-17T15:46:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.609+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 >> v1:172.21.15.64:6791/0 conn(0x7ff344135f70 legacy=0x7ff344137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 >> v1:172.21.15.100:6790/0 conn(0x7ff344135b50 legacy=0x7ff3441374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 10 monclient: _finish_hunting 0 2024-09-17T15:46:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 1 monclient: found mon.b 2024-09-17T15:46:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:06.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3441387c0 con 0x7ff3440a5520 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 10 monclient: _finish_auth 0 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 10 monclient: _check_auth_tickets 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff34acbd640 5 monclient: authenticate success, global_id 4340 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff34acbd640 10 monclient: _renew_subs 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff34acbd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff34acbd640 1 -- 172.21.15.64:0/2010866184 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff344141750 con 0x7ff3440a5520 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff34acbd640 10 monclient: _renew_subs 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff34acbd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff34acbd640 1 -- 172.21.15.64:0/2010866184 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff344139500 con 0x7ff3440a5520 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff32c0038c0 con 0x7ff3440a5520 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:06.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff32c003e30 con 0x7ff3440a5520 2024-09-17T15:46:06.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:06.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff342ffd640 4 set_mon_vals no callback set 2024-09-17T15:46:06.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:06.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.610+0000 7ff3417fa640 10 monclient: dump: 2024-09-17T15:46:06.636 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.636 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.636 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.636 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.636 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.611+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7ff32c003fd0 con 0x7ff3440a5520 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.611+0000 7ff34acbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:06.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.611+0000 7ff34acbd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:06.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.611+0000 7ff34acbd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:06.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.611+0000 7ff34acbd640 1 -- 172.21.15.64:0/2010866184 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff328005080 con 0x7ff3440a5520 2024-09-17T15:46:06.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.611+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7ff32c04eae0 con 0x7ff3440a5520 2024-09-17T15:46:06.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.611+0000 7ff343fff640 10 monclient: get_auth_request con 0x7ff324040dd0 auth_method 0 2024-09-17T15:46:06.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.615+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7ff34413a280 con 0x7ff3440a5520 2024-09-17T15:46:06.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.615+0000 7ff3417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:06.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.615+0000 7ff3417fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:06.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.797+0000 7ff34acbd640 1 -- 172.21.15.64:0/2010866184 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff328000b90 con 0x7ff324040dd0 2024-09-17T15:46:06.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.798+0000 7ff3417fa640 1 -- 172.21.15.64:0/2010866184 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34213 (unknown 2965378022 0 3364237253) 0x7ff328000b90 con 0x7ff324040dd0 2024-09-17T15:46:06.800 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:46:06.800 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-17T15:46:06.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.800+0000 7ff34acbd640 1 -- 172.21.15.64:0/2010866184 >> v1:172.21.15.100:6812/4210963484 conn(0x7ff324040dd0 legacy=0x7ff324043290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.800+0000 7ff34acbd640 10 monclient: shutdown 2024-09-17T15:46:06.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.800+0000 7ff34acbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:06.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.800+0000 7ff34acbd640 1 -- 172.21.15.64:0/2010866184 >> v1:172.21.15.64:6790/0 conn(0x7ff3440a5520 legacy=0x7ff344136dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.800+0000 7ff34acbd640 1 -- 172.21.15.64:0/2010866184 shutdown_connections 2024-09-17T15:46:06.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.800+0000 7ff34acbd640 1 -- 172.21.15.64:0/2010866184 >> 172.21.15.64:0/2010866184 conn(0x7ff34406db30 msgr2=0x7ff344133dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:06.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.800+0000 7ff34acbd640 1 -- 172.21.15.64:0/2010866184 shutdown_connections 2024-09-17T15:46:06.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.800+0000 7ff34acbd640 1 -- 172.21.15.64:0/2010866184 wait complete. 2024-09-17T15:46:06.812 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-17T15:46:05.833502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163244,"kb_used_data":1052,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067796,"statfs":{"total":579820584960,"available":579653423104,"internally_reserved":0,"allocated":1077248,"data_stored":1360066,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.906091"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952807+0000","last_change":"2024-09-17T15:46:03.953098+0000","last_active":"2024-09-17T15:46:03.952807+0000","last_peered":"2024-09-17T15:46:03.952807+0000","last_clean":"2024-09-17T15:46:03.952807+0000","last_became_active":"2024-09-17T15:46:02.372737+0000","last_became_peered":"2024-09-17T15:46:02.372737+0000","last_unstale":"2024-09-17T15:46:03.952807+0000","last_undegraded":"2024-09-17T15:46:03.952807+0000","last_fullsized":"2024-09-17T15:46:03.952807+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:04:26.679936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057658000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.954285+0000","last_change":"2024-09-17T15:46:03.954609+0000","last_active":"2024-09-17T15:46:03.954285+0000","last_peered":"2024-09-17T15:46:03.954285+0000","last_clean":"2024-09-17T15:46:03.954285+0000","last_became_active":"2024-09-17T15:46:01.943967+0000","last_became_peered":"2024-09-17T15:46:01.943967+0000","last_unstale":"2024-09-17T15:46:03.954285+0000","last_undegraded":"2024-09-17T15:46:03.954285+0000","last_fullsized":"2024-09-17T15:46:03.954285+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:44:26.433116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058926900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952760+0000","last_change":"2024-09-17T15:46:03.953301+0000","last_active":"2024-09-17T15:46:03.952760+0000","last_peered":"2024-09-17T15:46:03.952760+0000","last_clean":"2024-09-17T15:46:03.952760+0000","last_became_active":"2024-09-17T15:46:01.937233+0000","last_became_peered":"2024-09-17T15:46:01.937233+0000","last_unstale":"2024-09-17T15:46:03.952760+0000","last_undegraded":"2024-09-17T15:46:03.952760+0000","last_fullsized":"2024-09-17T15:46:03.952760+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:36:15.542838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001175981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.954344+0000","last_change":"2024-09-17T15:46:03.954824+0000","last_active":"2024-09-17T15:46:03.954344+0000","last_peered":"2024-09-17T15:46:03.954344+0000","last_clean":"2024-09-17T15:46:03.954344+0000","last_became_active":"2024-09-17T15:46:01.941419+0000","last_became_peered":"2024-09-17T15:46:01.941419+0000","last_unstale":"2024-09-17T15:46:03.954344+0000","last_undegraded":"2024-09-17T15:46:03.954344+0000","last_fullsized":"2024-09-17T15:46:03.954344+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:53:39.894895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077909099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952714+0000","last_change":"2024-09-17T15:46:03.952714+0000","last_active":"2024-09-17T15:46:03.952714+0000","last_peered":"2024-09-17T15:46:03.952714+0000","last_clean":"2024-09-17T15:46:03.952714+0000","last_became_active":"2024-09-17T15:46:01.941164+0000","last_became_peered":"2024-09-17T15:46:01.941164+0000","last_unstale":"2024-09-17T15:46:03.952714+0000","last_undegraded":"2024-09-17T15:46:03.952714+0000","last_fullsized":"2024-09-17T15:46:03.952714+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:07:59.957722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051738399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.954670+0000","last_change":"2024-09-17T15:46:03.954913+0000","last_active":"2024-09-17T15:46:03.954670+0000","last_peered":"2024-09-17T15:46:03.954670+0000","last_clean":"2024-09-17T15:46:03.954670+0000","last_became_active":"2024-09-17T15:46:01.941188+0000","last_became_peered":"2024-09-17T15:46:01.941188+0000","last_unstale":"2024-09-17T15:46:03.954670+0000","last_undegraded":"2024-09-17T15:46:03.954670+0000","last_fullsized":"2024-09-17T15:46:03.954670+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:25:32.291552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087819899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.953563+0000","last_change":"2024-09-17T15:46:03.953722+0000","last_active":"2024-09-17T15:46:03.953563+0000","last_peered":"2024-09-17T15:46:03.953563+0000","last_clean":"2024-09-17T15:46:03.953563+0000","last_became_active":"2024-09-17T15:46:01.941223+0000","last_became_peered":"2024-09-17T15:46:01.941223+0000","last_unstale":"2024-09-17T15:46:03.953563+0000","last_undegraded":"2024-09-17T15:46:03.953563+0000","last_fullsized":"2024-09-17T15:46:03.953563+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:18:15.696629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012017600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952412+0000","last_change":"2024-09-17T15:46:03.952788+0000","last_active":"2024-09-17T15:46:03.952412+0000","last_peered":"2024-09-17T15:46:03.952412+0000","last_clean":"2024-09-17T15:46:03.952412+0000","last_became_active":"2024-09-17T15:46:01.937420+0000","last_became_peered":"2024-09-17T15:46:01.937420+0000","last_unstale":"2024-09-17T15:46:03.952412+0000","last_undegraded":"2024-09-17T15:46:03.952412+0000","last_fullsized":"2024-09-17T15:46:03.952412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:03:18.015826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065159300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'72","reported_seq":121,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952920+0000","last_change":"2024-09-17T15:45:56.919389+0000","last_active":"2024-09-17T15:46:03.952920+0000","last_peered":"2024-09-17T15:46:03.952920+0000","last_clean":"2024-09-17T15:46:03.952920+0000","last_became_active":"2024-09-17T15:45:56.908629+0000","last_became_peered":"2024-09-17T15:45:56.908629+0000","last_unstale":"2024-09-17T15:46:03.952920+0000","last_undegraded":"2024-09-17T15:46:03.952920+0000","last_fullsized":"2024-09-17T15:46:03.952920+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_clean_scrub_stamp":"2024-09-17T15:45:53.875287+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:44:47.696204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":1057856,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":51050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":51069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":196608,"data_stored":575850,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":51050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":51069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":579978,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:46:06.813 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T15:46:06.813 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T15:46:06.813 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-17T15:46:06.813 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-17T15:46:06.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.911+0000 7f9958257640 10 monclient: get_monmap_and_config 2024-09-17T15:46:06.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.911+0000 7f9958257640 10 monclient: build_initial_monmap 2024-09-17T15:46:06.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.911+0000 7f9958257640 10 monclient: monmap: 2024-09-17T15:46:06.913 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:06.913 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.913 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:06.912039+0000 2024-09-17T15:46:06.913 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:06.912039+0000 2024-09-17T15:46:06.913 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:06.913 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.914 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:06.914 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:06.914 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:06.914 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:06.914 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:06.914 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:06.914 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:06.914 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:06.914 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:06.914 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.911+0000 7f9958257640 1 Processor -- start 2024-09-17T15:46:06.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.911+0000 7f9958257640 1 -- start start 2024-09-17T15:46:06.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9958257640 10 monclient: init 2024-09-17T15:46:06.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9958257640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:06.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9958257640 10 monclient: _add_conns ranks=[0,4,3,7,8,5,2,6,1] 2024-09-17T15:46:06.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9958257640 10 monclient(hunting): picked mon.noname-a con 0x7f99500a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:06.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9958257640 10 monclient(hunting): picked mon.noname-c con 0x7f9950135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:06.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9958257640 10 monclient(hunting): picked mon.noname-g con 0x7f9950135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9958257640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f995009d6f0 con 0x7f99500a5520 2024-09-17T15:46:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9958257640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f995009d9d0 con 0x7f9950135f70 2024-09-17T15:46:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9958257640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f995009dcb0 con 0x7f9950135b50 2024-09-17T15:46:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9958257640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9958257640 10 monclient(hunting): authenticate will time out at 764.587533s 2024-09-17T15:46:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f99557f2640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f9950135b50 0x7f995013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:36846/0 (socket says 172.21.15.64:36846) 2024-09-17T15:46:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9955ff3640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f99500a5520 0x7f99500a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:58994/0 (socket says 172.21.15.64:58994) 2024-09-17T15:46:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f99557f2640 1 -- 172.21.15.64:0/3811911499 learned_addr learned my addr 172.21.15.64:0/3811911499 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9954ff1640 1 -- 172.21.15.64:0/3811911499 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2633187030 0 0) 0x7f995009d6f0 con 0x7f99500a5520 2024-09-17T15:46:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9954ff1640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9954ff1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9954ff1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9954ff1640 10 monclient(hunting): my global_id is 4320 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9954ff1640 1 -- 172.21.15.64:0/3811911499 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f993c003520 con 0x7f99500a5520 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9954ff1640 1 -- 172.21.15.64:0/3811911499 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2488000722 0 0) 0x7f995009dcb0 con 0x7f9950135b50 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9954ff1640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9954ff1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9954ff1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9954ff1640 10 monclient(hunting): my global_id is 4423 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.912+0000 7f9954ff1640 1 -- 172.21.15.64:0/3811911499 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f995009d6f0 con 0x7f9950135b50 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 1 -- 172.21.15.64:0/3811911499 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1957918135 0 0) 0x7f995009d9d0 con 0x7f9950135f70 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient(hunting): my global_id is 4395 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 1 -- 172.21.15.64:0/3811911499 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f995009dcb0 con 0x7f9950135f70 2024-09-17T15:46:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 1 -- 172.21.15.64:0/3811911499 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2150338714 0 0) 0x7f993c003520 con 0x7f99500a5520 2024-09-17T15:46:06.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 1 -- 172.21.15.64:0/3811911499 >> v1:172.21.15.64:6791/0 conn(0x7f9950135b50 legacy=0x7f995013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 1 -- 172.21.15.64:0/3811911499 >> v1:172.21.15.100:6790/0 conn(0x7f9950135f70 legacy=0x7f99501415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient: _finish_hunting 0 2024-09-17T15:46:06.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 1 monclient: found mon.noname-a 2024-09-17T15:46:06.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:06.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:46:06.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 1 -- 172.21.15.64:0/3811911499 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f995009ccd0 con 0x7f99500a5520 2024-09-17T15:46:06.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient: _finish_auth 0 2024-09-17T15:46:06.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient: _check_auth_tickets 2024-09-17T15:46:06.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:06.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 1 -- 172.21.15.64:0/3811911499 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f994c001830 con 0x7f99500a5520 2024-09-17T15:46:06.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9958257640 5 monclient: authenticate success, global_id 4320 2024-09-17T15:46:06.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9958257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient: dump: 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.925 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.926 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9958257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 1 -- 172.21.15.64:0/3811911499 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f994c001a50 con 0x7f99500a5520 2024-09-17T15:46:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 1 -- 172.21.15.64:0/3811911499 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f994c004b20 con 0x7f99500a5520 2024-09-17T15:46:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9958257640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9958257640 4 set_mon_vals no callback set 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9954ff1640 10 monclient: dump: 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.927 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.928 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9958257640 10 monclient: shutdown 2024-09-17T15:46:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9958257640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9958257640 1 -- 172.21.15.64:0/3811911499 >> v1:172.21.15.64:6789/0 conn(0x7f99500a5520 legacy=0x7f99500a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9958257640 1 -- 172.21.15.64:0/3811911499 shutdown_connections 2024-09-17T15:46:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.913+0000 7f9958257640 1 -- 172.21.15.64:0/3811911499 >> 172.21.15.64:0/3811911499 conn(0x7f995006db30 msgr2=0x7f99500ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.914+0000 7f9958257640 1 -- 172.21.15.64:0/3811911499 shutdown_connections 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.914+0000 7f9958257640 1 -- 172.21.15.64:0/3811911499 wait complete. 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.914+0000 7f9958257640 10 monclient: build_initial_monmap 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.914+0000 7f9958257640 10 monclient: monmap: 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:06.929 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.914+0000 7f9958257640 1 Processor -- start 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.914+0000 7f9958257640 1 -- start start 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.914+0000 7f9958257640 10 monclient: init 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f9958257640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f9958257640 10 monclient: _add_conns ranks=[1,8,2,3,0,6,5,4,7] 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f9958257640 10 monclient(hunting): picked mon.noname-f con 0x7f99500a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f9958257640 10 monclient(hunting): picked mon.noname-e con 0x7f9950135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f9958257640 10 monclient(hunting): picked mon.noname-b con 0x7f9950135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:06.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f9958257640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99500a8fa0 con 0x7f99500a5520 2024-09-17T15:46:06.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f9958257640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9950138a30 con 0x7f9950135f70 2024-09-17T15:46:06.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f9958257640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9950137d60 con 0x7f9950135b50 2024-09-17T15:46:06.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f9958257640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:06.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f9958257640 10 monclient(hunting): authenticate will time out at 764.590406s 2024-09-17T15:46:06.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99557f2640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f9950135b50 0x7f9950137650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38660/0 (socket says 172.21.15.64:38660) 2024-09-17T15:46:06.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99557f2640 1 -- 172.21.15.64:0/2772749130 learned_addr learned my addr 172.21.15.64:0/2772749130 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:06.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2258279779 0 0) 0x7f9950138a30 con 0x7f9950135f70 2024-09-17T15:46:06.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 10 monclient(hunting): my global_id is 4349 2024-09-17T15:46:06.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9934003520 con 0x7f9950135f70 2024-09-17T15:46:06.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 284857349 0 0) 0x7f9950137d60 con 0x7f9950135b50 2024-09-17T15:46:06.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 10 monclient(hunting): my global_id is 4355 2024-09-17T15:46:06.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9950138a30 con 0x7f9950135b50 2024-09-17T15:46:06.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3645923905 0 0) 0x7f99500a8fa0 con 0x7f99500a5520 2024-09-17T15:46:06.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:06.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 10 monclient(hunting): my global_id is 4393 2024-09-17T15:46:06.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9950137d60 con 0x7f99500a5520 2024-09-17T15:46:06.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.915+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 47581880 0 0) 0x7f9934003520 con 0x7f9950135f70 2024-09-17T15:46:06.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99500a8fa0 con 0x7f9950135f70 2024-09-17T15:46:06.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9948001d70 con 0x7f9950135f70 2024-09-17T15:46:06.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:06.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:06.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient(hunting): dump: 2024-09-17T15:46:06.933 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.933 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.933 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.933 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.933 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.933 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.933 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.933 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.933 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 592207324 0 0) 0x7f9950138a30 con 0x7f9950135b50 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9934003520 con 0x7f9950135b50 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9940001a50 con 0x7f9950135b50 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient(hunting): dump: 2024-09-17T15:46:06.934 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.935 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.935 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.935 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.935 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.935 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.935 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.935 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.935 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.935 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.935 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.935 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.935 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.935 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2540684340 0 0) 0x7f9950137d60 con 0x7f99500a5520 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9950138a30 con 0x7f99500a5520 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f994c0017f0 con 0x7f99500a5520 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient(hunting): dump: 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.936 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.937 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.937 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.937 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.937 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.937 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.937 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.937 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.937 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.937 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.937 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2470079187 0 0) 0x7f99500a8fa0 con 0x7f9950135f70 2024-09-17T15:46:06.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 >> v1:172.21.15.64:6793/0 conn(0x7f9950135b50 legacy=0x7f9950137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 >> v1:172.21.15.100:6789/0 conn(0x7f99500a5520 legacy=0x7f9950136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 monclient: found mon.b 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f995013ad90 con 0x7f9950135f70 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f9958257640 5 monclient: authenticate success, global_id 4349 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f9958257640 10 monclient: _renew_subs 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f9958257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f9958257640 1 -- 172.21.15.64:0/2772749130 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f995013a4f0 con 0x7f9950135f70 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f9958257640 10 monclient: _renew_subs 2024-09-17T15:46:06.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f9958257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f9958257640 1 -- 172.21.15.64:0/2772749130 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9950139800 con 0x7f9950135f70 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9948002010 con 0x7f9950135f70 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9948003f80 con 0x7f9950135f70 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f9947fff640 4 set_mon_vals no callback set 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.916+0000 7f99467fc640 10 monclient: dump: 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:06.939 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.917+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f9948010860 con 0x7f9950135f70 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.917+0000 7f9958257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.917+0000 7f9958257640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:06.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.917+0000 7f9958257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:06.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.917+0000 7f9958257640 1 -- 172.21.15.64:0/2772749130 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9938005080 con 0x7f9950135f70 2024-09-17T15:46:06.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.917+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f994804d9f0 con 0x7f9950135f70 2024-09-17T15:46:06.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.917+0000 7f9955ff3640 10 monclient: get_auth_request con 0x7f9934040c60 auth_method 0 2024-09-17T15:46:06.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.921+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f99480166f0 con 0x7f9950135f70 2024-09-17T15:46:06.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.921+0000 7f99467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:06.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:06.921+0000 7f99467fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:07.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.143+0000 7f9958257640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-17T15:46:07.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.143+0000 7f9958257640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-17T15:46:07.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.143+0000 7f9958257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:07.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.143+0000 7f9958257640 1 -- 172.21.15.64:0/2772749130 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9938005370 con 0x7f9950135f70 2024-09-17T15:46:07.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.144+0000 7f99467fc640 1 -- 172.21.15.64:0/2772749130 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f994801aad0 con 0x7f9950135f70 2024-09-17T15:46:07.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.144+0000 7f99467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-17T15:46:07.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.144+0000 7f99467fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:07.146 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:46:07.146 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T15:46:07.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.146+0000 7f9958257640 1 -- 172.21.15.64:0/2772749130 >> v1:172.21.15.100:6812/4210963484 conn(0x7f9934040c60 legacy=0x7f9934043120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.146+0000 7f9958257640 10 monclient: shutdown 2024-09-17T15:46:07.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.146+0000 7f9958257640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:07.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.146+0000 7f9958257640 1 -- 172.21.15.64:0/2772749130 >> v1:172.21.15.64:6790/0 conn(0x7f9950135f70 legacy=0x7f99501d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.146+0000 7f9958257640 1 -- 172.21.15.64:0/2772749130 shutdown_connections 2024-09-17T15:46:07.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.146+0000 7f9958257640 1 -- 172.21.15.64:0/2772749130 >> 172.21.15.64:0/2772749130 conn(0x7f995006db30 msgr2=0x7f99500a4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:07.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.146+0000 7f9958257640 1 -- 172.21.15.64:0/2772749130 shutdown_connections 2024-09-17T15:46:07.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.146+0000 7f9958257640 1 -- 172.21.15.64:0/2772749130 wait complete. 2024-09-17T15:46:07.158 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-17T15:46:07.159 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-17T15:46:07.171 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-17T15:46:07.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T15:46:07.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.270+0000 7f3e1ba73640 10 monclient: get_monmap_and_config 2024-09-17T15:46:07.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.270+0000 7f3e1ba73640 10 monclient: build_initial_monmap 2024-09-17T15:46:07.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.270+0000 7f3e1ba73640 10 monclient: monmap: 2024-09-17T15:46:07.272 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:07.271296+0000 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:07.271296+0000 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:07.273 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:07.274 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1ba73640 1 Processor -- start 2024-09-17T15:46:07.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1ba73640 1 -- start start 2024-09-17T15:46:07.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1ba73640 10 monclient: init 2024-09-17T15:46:07.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1ba73640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:07.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1ba73640 10 monclient: _add_conns ranks=[2,3,8,0,7,6,5,1,4] 2024-09-17T15:46:07.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1ba73640 10 monclient(hunting): picked mon.noname-b con 0x7f3e140a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:07.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1ba73640 10 monclient(hunting): picked mon.noname-g con 0x7f3e14135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:07.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1ba73640 10 monclient(hunting): picked mon.noname-e con 0x7f3e14135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:07.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1ba73640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1409d6f0 con 0x7f3e140a5520 2024-09-17T15:46:07.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1ba73640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1409d9d0 con 0x7f3e14135b50 2024-09-17T15:46:07.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1ba73640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1409dcb0 con 0x7f3e14135f70 2024-09-17T15:46:07.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1ba73640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:07.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1ba73640 10 monclient(hunting): authenticate will time out at 764.946787s 2024-09-17T15:46:07.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1980f640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f3e140a5520 0x7f3e140a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46826/0 (socket says 172.21.15.64:46826) 2024-09-17T15:46:07.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1980f640 1 -- 172.21.15.64:0/3068940662 learned_addr learned my addr 172.21.15.64:0/3068940662 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:07.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.271+0000 7f3e1a010640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f3e14135f70 0x7f3e141415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38674/0 (socket says 172.21.15.64:38674) 2024-09-17T15:46:07.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 -- 172.21.15.64:0/3068940662 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 757137449 0 0) 0x7f3e1409d6f0 con 0x7f3e140a5520 2024-09-17T15:46:07.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient(hunting): my global_id is 4358 2024-09-17T15:46:07.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 -- 172.21.15.64:0/3068940662 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e00003520 con 0x7f3e140a5520 2024-09-17T15:46:07.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 -- 172.21.15.64:0/3068940662 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422745790 0 0) 0x7f3e1409dcb0 con 0x7f3e14135f70 2024-09-17T15:46:07.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient(hunting): my global_id is 4364 2024-09-17T15:46:07.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 -- 172.21.15.64:0/3068940662 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e1409d6f0 con 0x7f3e14135f70 2024-09-17T15:46:07.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 -- 172.21.15.64:0/3068940662 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1110699741 0 0) 0x7f3e1409d9d0 con 0x7f3e14135b50 2024-09-17T15:46:07.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient(hunting): my global_id is 4404 2024-09-17T15:46:07.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 -- 172.21.15.64:0/3068940662 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e1409dcb0 con 0x7f3e14135b50 2024-09-17T15:46:07.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 -- 172.21.15.64:0/3068940662 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3986078737 0 0) 0x7f3e00003520 con 0x7f3e140a5520 2024-09-17T15:46:07.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 -- 172.21.15.64:0/3068940662 >> v1:172.21.15.64:6793/0 conn(0x7f3e14135f70 legacy=0x7f3e141415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 -- 172.21.15.64:0/3068940662 >> v1:172.21.15.100:6790/0 conn(0x7f3e14135b50 legacy=0x7f3e1413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient: _finish_hunting 0 2024-09-17T15:46:07.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 monclient: found mon.noname-b 2024-09-17T15:46:07.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:07.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:46:07.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 -- 172.21.15.64:0/3068940662 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1409ccd0 con 0x7f3e140a5520 2024-09-17T15:46:07.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient: _finish_auth 0 2024-09-17T15:46:07.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient: _check_auth_tickets 2024-09-17T15:46:07.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:07.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1ba73640 5 monclient: authenticate success, global_id 4358 2024-09-17T15:46:07.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 -- 172.21.15.64:0/3068940662 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3e10001760 con 0x7f3e140a5520 2024-09-17T15:46:07.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1ba73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:07.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:07.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:07.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient: dump: 2024-09-17T15:46:07.284 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.285 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1ba73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 -- 172.21.15.64:0/3068940662 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3e10001980 con 0x7f3e140a5520 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1ba73640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1ba73640 4 set_mon_vals no callback set 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 1 -- 172.21.15.64:0/3068940662 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3e10004be0 con 0x7f3e140a5520 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1880d640 10 monclient: dump: 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.286 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1ba73640 10 monclient: shutdown 2024-09-17T15:46:07.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1ba73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:07.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.272+0000 7f3e1ba73640 1 -- 172.21.15.64:0/3068940662 >> v1:172.21.15.64:6790/0 conn(0x7f3e140a5520 legacy=0x7f3e140a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.273+0000 7f3e1ba73640 1 -- 172.21.15.64:0/3068940662 shutdown_connections 2024-09-17T15:46:07.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.273+0000 7f3e1ba73640 1 -- 172.21.15.64:0/3068940662 >> 172.21.15.64:0/3068940662 conn(0x7f3e1406db30 msgr2=0x7f3e140ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:07.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.273+0000 7f3e1ba73640 1 -- 172.21.15.64:0/3068940662 shutdown_connections 2024-09-17T15:46:07.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.273+0000 7f3e1ba73640 1 -- 172.21.15.64:0/3068940662 wait complete. 2024-09-17T15:46:07.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.273+0000 7f3e1ba73640 10 monclient: build_initial_monmap 2024-09-17T15:46:07.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.273+0000 7f3e1ba73640 10 monclient: monmap: 2024-09-17T15:46:07.288 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:07.288 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:07.288 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:07.288 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:07.288 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:07.288 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1ba73640 1 Processor -- start 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1ba73640 1 -- start start 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1ba73640 10 monclient: init 2024-09-17T15:46:07.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1ba73640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1ba73640 10 monclient: _add_conns ranks=[3,0,2,7,8,5,1,6,4] 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1ba73640 10 monclient(hunting): picked mon.noname-g con 0x7f3e140a5520 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1ba73640 10 monclient(hunting): picked mon.noname-a con 0x7f3e14135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1ba73640 10 monclient(hunting): picked mon.noname-b con 0x7f3e14135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1ba73640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1409ee90 con 0x7f3e14135b50 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1ba73640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e14134a90 con 0x7f3e14135f70 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1ba73640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e141346f0 con 0x7f3e140a5520 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1ba73640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1ba73640 10 monclient(hunting): authenticate will time out at 764.949548s 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1a010640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f3e14135f70 0x7f3e141d6360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46834/0 (socket says 172.21.15.64:46834) 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e1a010640 1 -- 172.21.15.64:0/1327710565 learned_addr learned my addr 172.21.15.64:0/1327710565 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3630585943 0 0) 0x7f3e0c0014b0 con 0x7f3e14135f70 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 10 monclient(hunting): my global_id is 4367 2024-09-17T15:46:07.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3df8003520 con 0x7f3e14135f70 2024-09-17T15:46:07.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2229389670 0 0) 0x7f3e1409ccd0 con 0x7f3e140a5520 2024-09-17T15:46:07.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 10 monclient(hunting): my global_id is 4413 2024-09-17T15:46:07.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e0c0014b0 con 0x7f3e140a5520 2024-09-17T15:46:07.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3065882925 0 0) 0x7f3e1409ee90 con 0x7f3e14135b50 2024-09-17T15:46:07.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 10 monclient(hunting): my global_id is 4329 2024-09-17T15:46:07.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.274+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e1409ccd0 con 0x7f3e14135b50 2024-09-17T15:46:07.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1297909573 0 0) 0x7f3e0c0014b0 con 0x7f3e140a5520 2024-09-17T15:46:07.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e1409ee90 con 0x7f3e140a5520 2024-09-17T15:46:07.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3e100017e0 con 0x7f3e140a5520 2024-09-17T15:46:07.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:07.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:46:07.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient(hunting): dump: 2024-09-17T15:46:07.292 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.292 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.292 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.292 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.292 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3145427253 0 0) 0x7f3df8003520 con 0x7f3e14135f70 2024-09-17T15:46:07.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e0c0014b0 con 0x7f3e14135f70 2024-09-17T15:46:07.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 307246925 0 0) 0x7f3e1409ccd0 con 0x7f3e14135b50 2024-09-17T15:46:07.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3df8003520 con 0x7f3e14135b50 2024-09-17T15:46:07.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3e0c001e60 con 0x7f3e14135f70 2024-09-17T15:46:07.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:07.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:07.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient(hunting): dump: 2024-09-17T15:46:07.294 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.294 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.294 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.294 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.294 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.294 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.294 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.295 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.295 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.295 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.295 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.295 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.295 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.295 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.295 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.295 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3e04001a10 con 0x7f3e14135b50 2024-09-17T15:46:07.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:07.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:07.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient(hunting): dump: 2024-09-17T15:46:07.296 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.296 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.296 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.296 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.296 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.296 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.296 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.296 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.297 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.297 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.297 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.297 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.297 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.297 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.297 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.297 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2515686717 0 0) 0x7f3e1409ee90 con 0x7f3e140a5520 2024-09-17T15:46:07.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 >> v1:172.21.15.64:6790/0 conn(0x7f3e14135f70 legacy=0x7f3e141d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 >> v1:172.21.15.64:6789/0 conn(0x7f3e14135b50 legacy=0x7f3e141ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:07.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 monclient: found mon.g 2024-09-17T15:46:07.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:07.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:46:07.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e140a8e90 con 0x7f3e140a5520 2024-09-17T15:46:07.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:07.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:07.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:07.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e1ba73640 5 monclient: authenticate success, global_id 4413 2024-09-17T15:46:07.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e1ba73640 10 monclient: _renew_subs 2024-09-17T15:46:07.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e1ba73640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:46:07.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e1ba73640 1 -- 172.21.15.64:0/1327710565 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3e141cef20 con 0x7f3e140a5520 2024-09-17T15:46:07.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e1ba73640 10 monclient: _renew_subs 2024-09-17T15:46:07.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e1ba73640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:46:07.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e1ba73640 1 -- 172.21.15.64:0/1327710565 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3e140a0830 con 0x7f3e140a5520 2024-09-17T15:46:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.3 v1:172.21.15.100:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3e10001980 con 0x7f3e140a5520 2024-09-17T15:46:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.3 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3e10004bd0 con 0x7f3e140a5520 2024-09-17T15:46:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:46:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0bfff640 4 set_mon_vals no callback set 2024-09-17T15:46:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.275+0000 7f3e0a7fc640 10 monclient: dump: 2024-09-17T15:46:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.299 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.276+0000 7f3e1ba73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.276+0000 7f3e1ba73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.276+0000 7f3e1ba73640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:46:07.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.276+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.3 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f3e10011920 con 0x7f3e140a5520 2024-09-17T15:46:07.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.276+0000 7f3e1ba73640 1 -- 172.21.15.64:0/1327710565 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dfc005080 con 0x7f3e140a5520 2024-09-17T15:46:07.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.276+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.3 v1:172.21.15.100:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f3e1004eb90 con 0x7f3e140a5520 2024-09-17T15:46:07.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.276+0000 7f3e1900e640 10 monclient: get_auth_request con 0x7f3df8040d70 auth_method 0 2024-09-17T15:46:07.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.281+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.3 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f3e140a8e90 con 0x7f3e140a5520 2024-09-17T15:46:07.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.281+0000 7f3e0a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:07.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.281+0000 7f3e0a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:07.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.468+0000 7f3e1ba73640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:46:07.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.468+0000 7f3e1ba73640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:46:07.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.468+0000 7f3e1ba73640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:46:07.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.468+0000 7f3e1ba73640 1 -- 172.21.15.64:0/1327710565 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3dfc005370 con 0x7f3e140a5520 2024-09-17T15:46:07.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.469+0000 7f3e0a7fc640 1 -- 172.21.15.64:0/1327710565 <== mon.3 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10953 (unknown 941590788 0 412723734) 0x7f3e141346f0 con 0x7f3e140a5520 2024-09-17T15:46:07.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.469+0000 7f3e0a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:46:07.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.469+0000 7f3e0a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:07.471 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:46:07.471 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":19,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","created":"2024-09-17T15:45:43.591977+0000","modified":"2024-09-17T15:46:03.940942+0000","last_up_change":"2024-09-17T15:45:55.895333+0000","last_in_change":"2024-09-17T15:45:45.218655+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T15:45:53.848072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T15:46:00.391921+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1d9b48f9-b24c-453f-a5e9-79d74f56118c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":796819178}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":796819178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":796819178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":796819178}]},"public_addr":"172.21.15.64:6808/796819178","cluster_addr":"172.21.15.64:6809/796819178","heartbeat_back_addr":"172.21.15.64:6811/796819178","heartbeat_front_addr":"172.21.15.64:6810/796819178","state":["exists","up"]},{"osd":1,"uuid":"6b8dcb8b-604c-46d3-a682-5ef3e93155d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6800","nonce":3889666033}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":3889666033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":3889666033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":3889666033}]},"public_addr":"172.21.15.64:6800/3889666033","cluster_addr":"172.21.15.64:6801/3889666033","heartbeat_back_addr":"172.21.15.64:6803/3889666033","heartbeat_front_addr":"172.21.15.64:6802/3889666033","state":["exists","up"]},{"osd":2,"uuid":"34a266c9-43fe-4e7b-ae85-df0115c5f618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":1825798494}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":1825798494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":1825798494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":1825798494}]},"public_addr":"172.21.15.64:6804/1825798494","cluster_addr":"172.21.15.64:6805/1825798494","heartbeat_back_addr":"172.21.15.64:6807/1825798494","heartbeat_front_addr":"172.21.15.64:6806/1825798494","state":["exists","up"]},{"osd":3,"uuid":"c8740bab-d046-474f-b6a4-84333eb1e094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6804","nonce":2791828047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6805","nonce":2791828047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6807","nonce":2791828047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6806","nonce":2791828047}]},"public_addr":"172.21.15.100:6804/2791828047","cluster_addr":"172.21.15.100:6805/2791828047","heartbeat_back_addr":"172.21.15.100:6807/2791828047","heartbeat_front_addr":"172.21.15.100:6806/2791828047","state":["exists","up"]},{"osd":4,"uuid":"08a1700b-63c5-4028-824f-dbd38931ddb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6808","nonce":4242749853}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6809","nonce":4242749853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6811","nonce":4242749853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6810","nonce":4242749853}]},"public_addr":"172.21.15.100:6808/4242749853","cluster_addr":"172.21.15.100:6809/4242749853","heartbeat_back_addr":"172.21.15.100:6811/4242749853","heartbeat_front_addr":"172.21.15.100:6810/4242749853","state":["exists","up"]},{"osd":5,"uuid":"81fa403f-39e9-44fd-8b1f-ef72fd7ba855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6800","nonce":4145568194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6801","nonce":4145568194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6803","nonce":4145568194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6802","nonce":4145568194}]},"public_addr":"172.21.15.100:6800/4145568194","cluster_addr":"172.21.15.100:6801/4145568194","heartbeat_back_addr":"172.21.15.100:6803/4145568194","heartbeat_front_addr":"172.21.15.100:6802/4145568194","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.703397+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.212068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.432455+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.332164+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.533990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.237115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:46:07.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.472+0000 7f3e1ba73640 1 -- 172.21.15.64:0/1327710565 >> v1:172.21.15.100:6812/4210963484 conn(0x7f3df8040d70 legacy=0x7f3df8043230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.472+0000 7f3e1ba73640 10 monclient: shutdown 2024-09-17T15:46:07.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.472+0000 7f3e1ba73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:07.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.472+0000 7f3e1ba73640 1 -- 172.21.15.64:0/1327710565 >> v1:172.21.15.100:6790/0 conn(0x7f3e140a5520 legacy=0x7f3e141ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.472+0000 7f3e1ba73640 1 -- 172.21.15.64:0/1327710565 shutdown_connections 2024-09-17T15:46:07.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.472+0000 7f3e1ba73640 1 -- 172.21.15.64:0/1327710565 >> 172.21.15.64:0/1327710565 conn(0x7f3e1406db30 msgr2=0x7f3e1412c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:07.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.472+0000 7f3e1ba73640 1 -- 172.21.15.64:0/1327710565 shutdown_connections 2024-09-17T15:46:07.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.472+0000 7f3e1ba73640 1 -- 172.21.15.64:0/1327710565 wait complete. 2024-09-17T15:46:07.484 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T15:45:53.848072+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-17T15:46:00.391921+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T15:46:07.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T15:46:07.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.588+0000 7f6125da1640 10 monclient: get_monmap_and_config 2024-09-17T15:46:07.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.588+0000 7f6125da1640 10 monclient: build_initial_monmap 2024-09-17T15:46:07.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.588+0000 7f6125da1640 10 monclient: monmap: 2024-09-17T15:46:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:07.589019+0000 2024-09-17T15:46:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:07.589019+0000 2024-09-17T15:46:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:07.590 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:07.591 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:07.591 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:07.591 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.588+0000 7f6125da1640 1 Processor -- start 2024-09-17T15:46:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.588+0000 7f6125da1640 1 -- start start 2024-09-17T15:46:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.588+0000 7f6125da1640 10 monclient: init 2024-09-17T15:46:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f6125da1640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f6125da1640 10 monclient: _add_conns ranks=[0,2,1,4,3,6,5,7,8] 2024-09-17T15:46:07.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f6125da1640 10 monclient(hunting): picked mon.noname-a con 0x7f61200a6320 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f6125da1640 10 monclient(hunting): picked mon.noname-b con 0x7f612009ce30 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f6125da1640 10 monclient(hunting): picked mon.noname-f con 0x7f6120138e90 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f6125da1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61200a5f10 con 0x7f61200a6320 2024-09-17T15:46:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f6125da1640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f612009d8d0 con 0x7f6120138e90 2024-09-17T15:46:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f6125da1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f612009db40 con 0x7f612009ce30 2024-09-17T15:46:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f6125da1640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f6125da1640 10 monclient(hunting): authenticate will time out at 765.264504s 2024-09-17T15:46:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f6124d9f640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f61200a6320 0x7f61200a6700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59008/0 (socket says 172.21.15.64:59008) 2024-09-17T15:46:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f6124d9f640 1 -- 172.21.15.64:0/3433685819 learned_addr learned my addr 172.21.15.64:0/3433685819 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f611f7fe640 1 -- 172.21.15.64:0/3433685819 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 26000677 0 0) 0x7f612009db40 con 0x7f612009ce30 2024-09-17T15:46:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f611f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f611f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f611f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f611f7fe640 10 monclient(hunting): my global_id is 4376 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f611f7fe640 1 -- 172.21.15.64:0/3433685819 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6110003520 con 0x7f612009ce30 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f611f7fe640 1 -- 172.21.15.64:0/3433685819 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2563941716 0 0) 0x7f61200a5f10 con 0x7f61200a6320 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f611f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f611f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f611f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f611f7fe640 10 monclient(hunting): my global_id is 4338 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.589+0000 7f611f7fe640 1 -- 172.21.15.64:0/3433685819 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f612009db40 con 0x7f61200a6320 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 1 -- 172.21.15.64:0/3433685819 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1793098749 0 0) 0x7f6110003520 con 0x7f612009ce30 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 1 -- 172.21.15.64:0/3433685819 >> v1:172.21.15.100:6789/0 conn(0x7f6120138e90 legacy=0x7f61201416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 1 -- 172.21.15.64:0/3433685819 >> v1:172.21.15.64:6789/0 conn(0x7f61200a6320 legacy=0x7f61200a6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 1 monclient: found mon.noname-b 2024-09-17T15:46:07.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 1 -- 172.21.15.64:0/3433685819 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612009de20 con 0x7f612009ce30 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 1 -- 172.21.15.64:0/3433685819 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f610c001830 con 0x7f612009ce30 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f6125da1640 5 monclient: authenticate success, global_id 4376 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 10 monclient: dump: 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.600 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f6125da1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:07.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 1 -- 172.21.15.64:0/3433685819 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f610c0019d0 con 0x7f612009ce30 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 1 -- 172.21.15.64:0/3433685819 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f610c003950 con 0x7f612009ce30 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f611f7fe640 10 monclient: dump: 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.602 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.603 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.603 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.603 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.603 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.603 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.603 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f6125da1640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:07.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f6125da1640 4 set_mon_vals no callback set 2024-09-17T15:46:07.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f6125da1640 10 monclient: shutdown 2024-09-17T15:46:07.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f6125da1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:07.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f6125da1640 1 -- 172.21.15.64:0/3433685819 >> v1:172.21.15.64:6790/0 conn(0x7f612009ce30 legacy=0x7f612013df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f6125da1640 1 -- 172.21.15.64:0/3433685819 shutdown_connections 2024-09-17T15:46:07.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f6125da1640 1 -- 172.21.15.64:0/3433685819 >> 172.21.15.64:0/3433685819 conn(0x7f61200ab920 msgr2=0x7f61200abd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f6125da1640 1 -- 172.21.15.64:0/3433685819 shutdown_connections 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.590+0000 7f6125da1640 1 -- 172.21.15.64:0/3433685819 wait complete. 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.591+0000 7f6125da1640 10 monclient: build_initial_monmap 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.591+0000 7f6125da1640 10 monclient: monmap: 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:07.604 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:07.605 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:07.605 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:07.605 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:07.605 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:07.605 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:07.605 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.591+0000 7f6125da1640 1 Processor -- start 2024-09-17T15:46:07.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.591+0000 7f6125da1640 1 -- start start 2024-09-17T15:46:07.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.591+0000 7f6125da1640 10 monclient: init 2024-09-17T15:46:07.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f6125da1640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:07.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f6125da1640 10 monclient: _add_conns ranks=[5,6,2,7,0,4,1,3,8] 2024-09-17T15:46:07.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f6125da1640 10 monclient(hunting): picked mon.noname-h con 0x7f612009ce30 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:07.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f6125da1640 10 monclient(hunting): picked mon.noname-d con 0x7f61200a6320 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f6125da1640 10 monclient(hunting): picked mon.noname-b con 0x7f6120138e90 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f6125da1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6120132380 con 0x7f6120138e90 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f6125da1640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61201307a0 con 0x7f612009ce30 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f6125da1640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6120132660 con 0x7f61200a6320 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f6125da1640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f6125da1640 10 monclient(hunting): authenticate will time out at 765.267440s 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f61255a0640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f6120138e90 0x7f61201d66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46846/0 (socket says 172.21.15.64:46846) 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f61255a0640 1 -- 172.21.15.64:0/55745086 learned_addr learned my addr 172.21.15.64:0/55745086 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611ffff640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f61200a6320 0x7f61201d2f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:49500/0 (socket says 172.21.15.64:49500) 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 398901445 0 0) 0x7f6120132380 con 0x7f6120138e90 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 10 monclient(hunting): my global_id is 4385 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6104003520 con 0x7f6120138e90 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 515221309 0 0) 0x7f6120132660 con 0x7f61200a6320 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 10 monclient(hunting): my global_id is 4317 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6120132380 con 0x7f61200a6320 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2383289905 0 0) 0x7f61201307a0 con 0x7f612009ce30 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 10 monclient(hunting): my global_id is 4325 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6120132660 con 0x7f612009ce30 2024-09-17T15:46:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 132465426 0 0) 0x7f6104003520 con 0x7f6120138e90 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61201307a0 con 0x7f6120138e90 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6118001e10 con 0x7f6120138e90 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.592+0000 7f611d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient(hunting): dump: 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.608 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 143968016 0 0) 0x7f6120132380 con 0x7f61200a6320 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6104003520 con 0x7f61200a6320 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f610c0039f0 con 0x7f61200a6320 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient(hunting): dump: 2024-09-17T15:46:07.609 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2902857816 0 0) 0x7f6120132660 con 0x7f612009ce30 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6120132380 con 0x7f612009ce30 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f61140013f0 con 0x7f612009ce30 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient(hunting): dump: 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.611 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.612 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.612 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.612 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.612 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.612 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.612 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.612 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.612 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.612 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.612 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2475155366 0 0) 0x7f61201307a0 con 0x7f6120138e90 2024-09-17T15:46:07.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 >> v1:172.21.15.64:6792/0 conn(0x7f61200a6320 legacy=0x7f61201d2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 >> v1:172.21.15.100:6791/0 conn(0x7f612009ce30 legacy=0x7f61201cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient: _finish_hunting 0 2024-09-17T15:46:07.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 1 monclient: found mon.b 2024-09-17T15:46:07.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:07.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:07.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6120138170 con 0x7f6120138e90 2024-09-17T15:46:07.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:46:07.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:46:07.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:07.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f6125da1640 5 monclient: authenticate success, global_id 4385 2024-09-17T15:46:07.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f6125da1640 10 monclient: _renew_subs 2024-09-17T15:46:07.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f6125da1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:07.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f6125da1640 1 -- 172.21.15.64:0/55745086 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f61201320a0 con 0x7f6120138e90 2024-09-17T15:46:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f6125da1640 10 monclient: _renew_subs 2024-09-17T15:46:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f6125da1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f6125da1640 1 -- 172.21.15.64:0/55745086 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f612012fc10 con 0x7f6120138e90 2024-09-17T15:46:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6118003c40 con 0x7f6120138e90 2024-09-17T15:46:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6118004230 con 0x7f6120138e90 2024-09-17T15:46:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611effd640 4 set_mon_vals no callback set 2024-09-17T15:46:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.593+0000 7f611d7fa640 10 monclient: dump: 2024-09-17T15:46:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.614 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.615 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.615 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.615 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.615 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.615 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.615 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.615 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.615 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.615 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.615 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.615 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.615 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.615 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.594+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f61180043d0 con 0x7f6120138e90 2024-09-17T15:46:07.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.594+0000 7f6125da1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:07.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.594+0000 7f6125da1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:07.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.594+0000 7f6125da1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:07.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.594+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f611804edb0 con 0x7f6120138e90 2024-09-17T15:46:07.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.594+0000 7f6125da1640 1 -- 172.21.15.64:0/55745086 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6100005080 con 0x7f6120138e90 2024-09-17T15:46:07.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.594+0000 7f6124d9f640 10 monclient: get_auth_request con 0x7f6104040cf0 auth_method 0 2024-09-17T15:46:07.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.599+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f61180179d0 con 0x7f6120138e90 2024-09-17T15:46:07.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.599+0000 7f611d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:07.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.599+0000 7f611d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:07.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.781+0000 7f6125da1640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-17T15:46:07.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.781+0000 7f6125da1640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-17T15:46:07.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.781+0000 7f6125da1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:07.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.781+0000 7f6125da1640 1 -- 172.21.15.64:0/55745086 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6100005b80 con 0x7f6120138e90 2024-09-17T15:46:07.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.781+0000 7f611d7fa640 1 -- 172.21.15.64:0/55745086 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) ==== 93+0+10 (unknown 4170632370 0 2170607528) 0x7f611801bd50 con 0x7f6120138e90 2024-09-17T15:46:07.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.781+0000 7f611d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-17T15:46:07.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.781+0000 7f611d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:07.784 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-09-17T15:46:07.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.784+0000 7f6125da1640 1 -- 172.21.15.64:0/55745086 >> v1:172.21.15.100:6812/4210963484 conn(0x7f6104040cf0 legacy=0x7f61040431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.784+0000 7f6125da1640 10 monclient: shutdown 2024-09-17T15:46:07.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.784+0000 7f6125da1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:07.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.784+0000 7f6125da1640 1 -- 172.21.15.64:0/55745086 >> v1:172.21.15.64:6790/0 conn(0x7f6120138e90 legacy=0x7f61201d66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.784+0000 7f6125da1640 1 -- 172.21.15.64:0/55745086 shutdown_connections 2024-09-17T15:46:07.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.784+0000 7f6125da1640 1 -- 172.21.15.64:0/55745086 >> 172.21.15.64:0/55745086 conn(0x7f61200ab920 msgr2=0x7f61201367c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:07.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.784+0000 7f6125da1640 1 -- 172.21.15.64:0/55745086 shutdown_connections 2024-09-17T15:46:07.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.784+0000 7f6125da1640 1 -- 172.21.15.64:0/55745086 wait complete. 2024-09-17T15:46:07.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-17T15:46:07.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.897+0000 7f448ce60640 10 monclient: get_monmap_and_config 2024-09-17T15:46:07.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.897+0000 7f448ce60640 10 monclient: build_initial_monmap 2024-09-17T15:46:07.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.897+0000 7f448ce60640 10 monclient: monmap: 2024-09-17T15:46:07.899 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:07.899 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.899 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:07.898236+0000 2024-09-17T15:46:07.899 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:07.898236+0000 2024-09-17T15:46:07.899 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:07.899 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.899 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:07.900 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:07.900 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:07.900 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:07.900 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:07.900 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:07.900 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:07.900 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:07.900 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:07.900 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f448ce60640 1 Processor -- start 2024-09-17T15:46:07.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f448ce60640 1 -- start start 2024-09-17T15:46:07.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f448ce60640 10 monclient: init 2024-09-17T15:46:07.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f448ce60640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:07.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f448ce60640 10 monclient: _add_conns ranks=[3,8,4,0,1,5,7,2,6] 2024-09-17T15:46:07.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f448ce60640 10 monclient(hunting): picked mon.noname-g con 0x7f44880a5600 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:07.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f448ce60640 10 monclient(hunting): picked mon.noname-e con 0x7f44881397e0 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:07.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f448ce60640 10 monclient(hunting): picked mon.noname-c con 0x7f448813f800 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:07.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f448ce60640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f448809dab0 con 0x7f44880a5600 2024-09-17T15:46:07.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f448ce60640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f448809dd90 con 0x7f448813f800 2024-09-17T15:46:07.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f448ce60640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f448809e070 con 0x7f44881397e0 2024-09-17T15:46:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f448ce60640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f448ce60640 10 monclient(hunting): authenticate will time out at 765.573691s 2024-09-17T15:46:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f447ffff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f44881397e0 0x7f448813e0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38676/0 (socket says 172.21.15.64:38676) 2024-09-17T15:46:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.898+0000 7f447ffff640 1 -- 172.21.15.64:0/312083903 learned_addr learned my addr 172.21.15.64:0/312083903 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 1 -- 172.21.15.64:0/312083903 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 642621726 0 0) 0x7f448809dd90 con 0x7f448813f800 2024-09-17T15:46:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient(hunting): my global_id is 4432 2024-09-17T15:46:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 1 -- 172.21.15.64:0/312083903 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4474003520 con 0x7f448813f800 2024-09-17T15:46:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 1 -- 172.21.15.64:0/312083903 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 392472699 0 0) 0x7f448809e070 con 0x7f44881397e0 2024-09-17T15:46:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient(hunting): my global_id is 4373 2024-09-17T15:46:07.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 1 -- 172.21.15.64:0/312083903 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f448809dd90 con 0x7f44881397e0 2024-09-17T15:46:07.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 1 -- 172.21.15.64:0/312083903 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3247345008 0 0) 0x7f4474003520 con 0x7f448813f800 2024-09-17T15:46:07.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 1 -- 172.21.15.64:0/312083903 >> v1:172.21.15.64:6793/0 conn(0x7f44881397e0 legacy=0x7f448813e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 1 -- 172.21.15.64:0/312083903 >> v1:172.21.15.100:6790/0 conn(0x7f44880a5600 legacy=0x7f44880a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:46:07.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 1 monclient: found mon.noname-c 2024-09-17T15:46:07.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:07.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:46:07.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 1 -- 172.21.15.64:0/312083903 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f448809ca00 con 0x7f448813f800 2024-09-17T15:46:07.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient: _finish_auth 0 2024-09-17T15:46:07.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:46:07.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:07.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 1 -- 172.21.15.64:0/312083903 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4480001bb0 con 0x7f448813f800 2024-09-17T15:46:07.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f448ce60640 5 monclient: authenticate success, global_id 4432 2024-09-17T15:46:07.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f448ce60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:07.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:07.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:07.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient: dump: 2024-09-17T15:46:07.910 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.910 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.910 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.910 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.910 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f448ce60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 1 -- 172.21.15.64:0/312083903 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4480001dd0 con 0x7f448813f800 2024-09-17T15:46:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:07.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f448ce60640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:07.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f448ce60640 4 set_mon_vals no callback set 2024-09-17T15:46:07.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 1 -- 172.21.15.64:0/312083903 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4480003cd0 con 0x7f448813f800 2024-09-17T15:46:07.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:07.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:07.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f4485d9b640 10 monclient: dump: 2024-09-17T15:46:07.912 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.912 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.912 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.912 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.912 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.912 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.912 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.913 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.913 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.913 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.913 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.913 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.913 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.913 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.913 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.913 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f448ce60640 10 monclient: shutdown 2024-09-17T15:46:07.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f448ce60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:07.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.899+0000 7f448ce60640 1 -- 172.21.15.64:0/312083903 >> v1:172.21.15.64:6791/0 conn(0x7f448813f800 legacy=0x7f4488141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.900+0000 7f448ce60640 1 -- 172.21.15.64:0/312083903 shutdown_connections 2024-09-17T15:46:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.900+0000 7f448ce60640 1 -- 172.21.15.64:0/312083903 >> 172.21.15.64:0/312083903 conn(0x7f448806dd40 msgr2=0x7f44880aba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.900+0000 7f448ce60640 1 -- 172.21.15.64:0/312083903 shutdown_connections 2024-09-17T15:46:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.900+0000 7f448ce60640 1 -- 172.21.15.64:0/312083903 wait complete. 2024-09-17T15:46:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.900+0000 7f448ce60640 10 monclient: build_initial_monmap 2024-09-17T15:46:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.900+0000 7f448ce60640 10 monclient: monmap: 2024-09-17T15:46:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:07.915 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:07.915 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:07.915 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:07.915 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:07.915 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:07.915 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:07.915 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:07.915 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448ce60640 1 Processor -- start 2024-09-17T15:46:07.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448ce60640 1 -- start start 2024-09-17T15:46:07.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448ce60640 10 monclient: init 2024-09-17T15:46:07.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448ce60640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:07.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448ce60640 10 monclient: _add_conns ranks=[0,1,7,4,3,8,2,5,6] 2024-09-17T15:46:07.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448ce60640 10 monclient(hunting): picked mon.noname-a con 0x7f44880a5600 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:07.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448ce60640 10 monclient(hunting): picked mon.noname-f con 0x7f44881397e0 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:07.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448ce60640 10 monclient(hunting): picked mon.noname-i con 0x7f448813f800 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:07.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448ce60640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4488138a30 con 0x7f44880a5600 2024-09-17T15:46:07.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448ce60640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44880a0f40 con 0x7f44881397e0 2024-09-17T15:46:07.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448ce60640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4488133000 con 0x7f448813f800 2024-09-17T15:46:07.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448ce60640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:07.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448ce60640 10 monclient(hunting): authenticate will time out at 765.576591s 2024-09-17T15:46:07.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448659c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f44880a5600 0x7f44881321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59018/0 (socket says 172.21.15.64:59018) 2024-09-17T15:46:07.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f448659c640 1 -- 172.21.15.64:0/3268782088 learned_addr learned my addr 172.21.15.64:0/3268782088 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:07.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 586875610 0 0) 0x7f4478001170 con 0x7f44880a5600 2024-09-17T15:46:07.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f447effd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f447effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f447effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f447effd640 10 monclient(hunting): my global_id is 4347 2024-09-17T15:46:07.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.901+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4468003520 con 0x7f44880a5600 2024-09-17T15:46:07.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1501968426 0 0) 0x7f44880a0f40 con 0x7f44881397e0 2024-09-17T15:46:07.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient(hunting): my global_id is 4411 2024-09-17T15:46:07.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4478001170 con 0x7f44881397e0 2024-09-17T15:46:07.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2969151248 0 0) 0x7f4488133000 con 0x7f448813f800 2024-09-17T15:46:07.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:07.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:07.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient(hunting): my global_id is 4372 2024-09-17T15:46:07.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44880a0f40 con 0x7f448813f800 2024-09-17T15:46:07.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3338429269 0 0) 0x7f4468003520 con 0x7f44880a5600 2024-09-17T15:46:07.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4488133000 con 0x7f44880a5600 2024-09-17T15:46:07.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4478001d40 con 0x7f44880a5600 2024-09-17T15:46:07.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:07.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:07.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient(hunting): dump: 2024-09-17T15:46:07.918 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.918 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.918 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:07.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2744433910 0 0) 0x7f4478001170 con 0x7f44881397e0 2024-09-17T15:46:07.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4468003520 con 0x7f44881397e0 2024-09-17T15:46:07.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 658725153 0 0) 0x7f44880a0f40 con 0x7f448813f800 2024-09-17T15:46:07.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4478001170 con 0x7f448813f800 2024-09-17T15:46:07.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1332124868 0 0) 0x7f4488133000 con 0x7f44880a5600 2024-09-17T15:46:07.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 >> v1:172.21.15.100:6792/0 conn(0x7f448813f800 legacy=0x7f44881d5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 >> v1:172.21.15.100:6789/0 conn(0x7f44881397e0 legacy=0x7f44881328f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:07.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:07.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 monclient: found mon.a 2024-09-17T15:46:07.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:07.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:07.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4488138670 con 0x7f44880a5600 2024-09-17T15:46:07.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient: _finish_auth 0 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f448ce60640 5 monclient: authenticate success, global_id 4347 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f448ce60640 10 monclient: _renew_subs 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f448ce60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f448ce60640 1 -- 172.21.15.64:0/3268782088 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f44880aa8a0 con 0x7f44880a5600 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f448ce60640 10 monclient: _renew_subs 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f448ce60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f448ce60640 1 -- 172.21.15.64:0/3268782088 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f448813dd20 con 0x7f44880a5600 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4478003a20 con 0x7f44880a5600 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4478003f50 con 0x7f44880a5600 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:07.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f448559a640 4 set_mon_vals no callback set 2024-09-17T15:46:07.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:07.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.902+0000 7f447effd640 10 monclient: dump: 2024-09-17T15:46:07.922 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:07.922 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:07.922 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.922 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:07.922 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:07.922 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:07.922 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:07.922 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:07.922 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:07.922 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.903+0000 7f448ce60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.903+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f44780040f0 con 0x7f44880a5600 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.903+0000 7f448ce60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.903+0000 7f448ce60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.903+0000 7f448ce60640 1 -- 172.21.15.64:0/3268782088 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f446c005080 con 0x7f44880a5600 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.903+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7f447804ed00 con 0x7f44880a5600 2024-09-17T15:46:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.904+0000 7f447ffff640 10 monclient: get_auth_request con 0x7f4468040890 auth_method 0 2024-09-17T15:46:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.908+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f4488138a30 con 0x7f44880a5600 2024-09-17T15:46:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.908+0000 7f447effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:07.908+0000 7f447effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:08.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.095+0000 7f448ce60640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-17T15:46:08.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.095+0000 7f448ce60640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-17T15:46:08.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.095+0000 7f448ce60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:08.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.095+0000 7f448ce60640 1 -- 172.21.15.64:0/3268782088 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f446c005c40 con 0x7f44880a5600 2024-09-17T15:46:08.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.095+0000 7f447effd640 1 -- 172.21.15.64:0/3268782088 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) ==== 92+0+10 (unknown 1398289234 0 265718887) 0x7f448809dab0 con 0x7f44880a5600 2024-09-17T15:46:08.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.095+0000 7f447effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-17T15:46:08.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.095+0000 7f447effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:08.098 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 8 2024-09-17T15:46:08.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.098+0000 7f448ce60640 1 -- 172.21.15.64:0/3268782088 >> v1:172.21.15.100:6812/4210963484 conn(0x7f4468040890 legacy=0x7f4468042d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.098+0000 7f448ce60640 10 monclient: shutdown 2024-09-17T15:46:08.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.098+0000 7f448ce60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:08.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.098+0000 7f448ce60640 1 -- 172.21.15.64:0/3268782088 >> v1:172.21.15.64:6789/0 conn(0x7f44880a5600 legacy=0x7f44881321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.098+0000 7f448ce60640 1 -- 172.21.15.64:0/3268782088 shutdown_connections 2024-09-17T15:46:08.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.098+0000 7f448ce60640 1 -- 172.21.15.64:0/3268782088 >> 172.21.15.64:0/3268782088 conn(0x7f448806dd40 msgr2=0x7f44881356f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:08.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.098+0000 7f448ce60640 1 -- 172.21.15.64:0/3268782088 shutdown_connections 2024-09-17T15:46:08.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.098+0000 7f448ce60640 1 -- 172.21.15.64:0/3268782088 wait complete. 2024-09-17T15:46:08.110 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-17T15:46:08.110 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:46:08.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.209+0000 7fc344d4b640 10 monclient: get_monmap_and_config 2024-09-17T15:46:08.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.209+0000 7fc344d4b640 10 monclient: build_initial_monmap 2024-09-17T15:46:08.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.209+0000 7fc344d4b640 10 monclient: monmap: 2024-09-17T15:46:08.211 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:08.211 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.211 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:08.210403+0000 2024-09-17T15:46:08.211 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:08.210403+0000 2024-09-17T15:46:08.211 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:08.211 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.211 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:08.212 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:08.212 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:08.212 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:08.212 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:08.212 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:08.212 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:08.212 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:08.212 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:08.212 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc344d4b640 1 Processor -- start 2024-09-17T15:46:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc344d4b640 1 -- start start 2024-09-17T15:46:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc344d4b640 10 monclient: init 2024-09-17T15:46:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc344d4b640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc344d4b640 10 monclient: _add_conns ranks=[5,6,4,3,7,1,8,2,0] 2024-09-17T15:46:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc344d4b640 10 monclient(hunting): picked mon.noname-h con 0x7fc3400a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:08.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc344d4b640 10 monclient(hunting): picked mon.noname-d con 0x7fc340135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc344d4b640 10 monclient(hunting): picked mon.noname-c con 0x7fc340135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc344d4b640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc34009d6f0 con 0x7fc340135f70 2024-09-17T15:46:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc344d4b640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc34009d9d0 con 0x7fc3400a5520 2024-09-17T15:46:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc344d4b640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc34009dcb0 con 0x7fc340135b50 2024-09-17T15:46:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc344d4b640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc344d4b640 10 monclient(hunting): authenticate will time out at 765.885741s 2024-09-17T15:46:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc335d9b640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fc340135b50 0x7fc34013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:49516/0 (socket says 172.21.15.64:49516) 2024-09-17T15:46:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc335d9b640 1 -- 172.21.15.64:0/10577905 learned_addr learned my addr 172.21.15.64:0/10577905 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.210+0000 7fc33ed9d640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fc340135f70 0x7fc3401415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:36866/0 (socket says 172.21.15.64:36866) 2024-09-17T15:46:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 1 -- 172.21.15.64:0/10577905 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 515781616 0 0) 0x7fc34009dcb0 con 0x7fc340135b50 2024-09-17T15:46:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient(hunting): my global_id is 4326 2024-09-17T15:46:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 1 -- 172.21.15.64:0/10577905 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc32c003520 con 0x7fc340135b50 2024-09-17T15:46:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 1 -- 172.21.15.64:0/10577905 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3490556963 0 0) 0x7fc34009d6f0 con 0x7fc340135f70 2024-09-17T15:46:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient(hunting): my global_id is 4441 2024-09-17T15:46:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 1 -- 172.21.15.64:0/10577905 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc34009dcb0 con 0x7fc340135f70 2024-09-17T15:46:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 1 -- 172.21.15.64:0/10577905 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1006383368 0 0) 0x7fc32c003520 con 0x7fc340135b50 2024-09-17T15:46:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 1 -- 172.21.15.64:0/10577905 >> v1:172.21.15.100:6791/0 conn(0x7fc3400a5520 legacy=0x7fc3400a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 1 -- 172.21.15.64:0/10577905 >> v1:172.21.15.64:6791/0 conn(0x7fc340135f70 legacy=0x7fc3401415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 1 monclient: found mon.noname-d 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 1 -- 172.21.15.64:0/10577905 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc34009ccd0 con 0x7fc340135b50 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient: _finish_auth 0 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient: _check_auth_tickets 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 1 -- 172.21.15.64:0/10577905 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc328001830 con 0x7fc340135b50 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc344d4b640 5 monclient: authenticate success, global_id 4326 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc344d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient: dump: 2024-09-17T15:46:08.223 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.224 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.224 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.224 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.224 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.224 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.224 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.224 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.224 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.224 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.224 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.224 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.224 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.224 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc344d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 1 -- 172.21.15.64:0/10577905 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc328001a50 con 0x7fc340135b50 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc344d4b640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc344d4b640 4 set_mon_vals no callback set 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 1 -- 172.21.15.64:0/10577905 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc328001d40 con 0x7fc340135b50 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc33dd9b640 10 monclient: dump: 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.225 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.226 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.226 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.226 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.226 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.226 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.226 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.226 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.226 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.226 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.226 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.226 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.226 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.226 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc344d4b640 10 monclient: shutdown 2024-09-17T15:46:08.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc344d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:08.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.211+0000 7fc344d4b640 1 -- 172.21.15.64:0/10577905 >> v1:172.21.15.64:6792/0 conn(0x7fc340135b50 legacy=0x7fc34013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.212+0000 7fc344d4b640 1 -- 172.21.15.64:0/10577905 shutdown_connections 2024-09-17T15:46:08.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.212+0000 7fc344d4b640 1 -- 172.21.15.64:0/10577905 >> 172.21.15.64:0/10577905 conn(0x7fc34006db30 msgr2=0x7fc3400ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:08.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.212+0000 7fc344d4b640 1 -- 172.21.15.64:0/10577905 shutdown_connections 2024-09-17T15:46:08.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.212+0000 7fc344d4b640 1 -- 172.21.15.64:0/10577905 wait complete. 2024-09-17T15:46:08.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.212+0000 7fc344d4b640 10 monclient: build_initial_monmap 2024-09-17T15:46:08.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.212+0000 7fc344d4b640 10 monclient: monmap: 2024-09-17T15:46:08.227 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:08.227 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:08.228 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:08.228 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:08.228 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:08.228 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:08.228 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:08.228 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:08.228 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:08.228 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:08.228 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:08.228 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:08.228 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:08.228 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc344d4b640 1 Processor -- start 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc344d4b640 1 -- start start 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc344d4b640 10 monclient: init 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc344d4b640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc344d4b640 10 monclient: _add_conns ranks=[8,3,4,6,5,2,1,0,7] 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc344d4b640 10 monclient(hunting): picked mon.noname-e con 0x7fc3400a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc344d4b640 10 monclient(hunting): picked mon.noname-g con 0x7fc340135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc344d4b640 10 monclient(hunting): picked mon.noname-c con 0x7fc340135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc344d4b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc340134f40 con 0x7fc340135b50 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc344d4b640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3400a3450 con 0x7fc340135f70 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc344d4b640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3400a1b50 con 0x7fc3400a5520 2024-09-17T15:46:08.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc344d4b640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc344d4b640 10 monclient(hunting): authenticate will time out at 765.888826s 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc33ed9d640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fc340135f70 0x7fc3401d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:36874/0 (socket says 172.21.15.64:36874) 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc33ed9d640 1 -- 172.21.15.64:0/59818045 learned_addr learned my addr 172.21.15.64:0/59818045 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.213+0000 7fc33e59c640 1 --1- 172.21.15.64:0/59818045 >> v1:172.21.15.64:6793/0 conn(0x7fc3400a5520 0x7fc34009fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38678/0 (socket says 172.21.15.64:38678) 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1764022232 0 0) 0x7fc3400a1b50 con 0x7fc3400a5520 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): my global_id is 4382 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc320003520 con 0x7fc3400a5520 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1955066385 0 0) 0x7fc3400a3450 con 0x7fc340135f70 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): my global_id is 4450 2024-09-17T15:46:08.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3400a1b50 con 0x7fc340135f70 2024-09-17T15:46:08.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4124390653 0 0) 0x7fc34009ccd0 con 0x7fc340135b50 2024-09-17T15:46:08.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): my global_id is 4431 2024-09-17T15:46:08.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3400a3450 con 0x7fc340135b50 2024-09-17T15:46:08.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1340703223 0 0) 0x7fc320003520 con 0x7fc3400a5520 2024-09-17T15:46:08.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc34009ccd0 con 0x7fc3400a5520 2024-09-17T15:46:08.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1233617678 0 0) 0x7fc3400a1b50 con 0x7fc340135f70 2024-09-17T15:46:08.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc320003520 con 0x7fc340135f70 2024-09-17T15:46:08.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc330001af0 con 0x7fc3400a5520 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): dump: 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.232 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc338001c40 con 0x7fc340135f70 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): dump: 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.233 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1055784181 0 0) 0x7fc3400a3450 con 0x7fc340135b50 2024-09-17T15:46:08.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3400a1b50 con 0x7fc340135b50 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc3280017f0 con 0x7fc340135b50 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient(hunting): dump: 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.235 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.236 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.236 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.236 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.236 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.236 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.236 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1205529266 0 0) 0x7fc34009ccd0 con 0x7fc3400a5520 2024-09-17T15:46:08.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 >> v1:172.21.15.64:6791/0 conn(0x7fc340135f70 legacy=0x7fc3401d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 >> v1:172.21.15.100:6790/0 conn(0x7fc340135b50 legacy=0x7fc3400a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:08.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 monclient: found mon.e 2024-09-17T15:46:08.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:08.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:08.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3400a3730 con 0x7fc3400a5520 2024-09-17T15:46:08.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:08.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:08.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:08.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc344d4b640 5 monclient: authenticate success, global_id 4382 2024-09-17T15:46:08.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc344d4b640 10 monclient: _renew_subs 2024-09-17T15:46:08.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc344d4b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:08.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc344d4b640 1 -- 172.21.15.64:0/59818045 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc3400a0ad0 con 0x7fc3400a5520 2024-09-17T15:46:08.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc344d4b640 10 monclient: _renew_subs 2024-09-17T15:46:08.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc344d4b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:08.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc344d4b640 1 -- 172.21.15.64:0/59818045 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3400a1f90 con 0x7fc3400a5520 2024-09-17T15:46:08.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3300038c0 con 0x7fc3400a5520 2024-09-17T15:46:08.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:08.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc330003e30 con 0x7fc3400a5520 2024-09-17T15:46:08.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:08.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc33d59a640 4 set_mon_vals no callback set 2024-09-17T15:46:08.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:08.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.214+0000 7fc3377fe640 10 monclient: dump: 2024-09-17T15:46:08.238 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.238 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.238 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.238 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.238 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.238 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.239 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.239 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.239 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.239 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.239 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.239 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.239 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.239 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.239 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.239 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.215+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fc330003fd0 con 0x7fc3400a5520 2024-09-17T15:46:08.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.215+0000 7fc344d4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:08.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.215+0000 7fc344d4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:08.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.215+0000 7fc344d4b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:08.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.215+0000 7fc344d4b640 1 -- 172.21.15.64:0/59818045 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc324005080 con 0x7fc3400a5520 2024-09-17T15:46:08.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.216+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1608679582 0 0) 0x7fc33004ea00 con 0x7fc3400a5520 2024-09-17T15:46:08.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.216+0000 7fc335d9b640 10 monclient: get_auth_request con 0x7fc320040c80 auth_method 0 2024-09-17T15:46:08.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.220+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mon.8 v1:172.21.15.64:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fc330017740 con 0x7fc3400a5520 2024-09-17T15:46:08.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.220+0000 7fc3377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:08.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.220+0000 7fc3377fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:08.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.404+0000 7fc344d4b640 1 -- 172.21.15.64:0/59818045 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc324000b90 con 0x7fc320040c80 2024-09-17T15:46:08.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.405+0000 7fc3377fe640 1 -- 172.21.15.64:0/59818045 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34189 (unknown 2965378022 0 1866897357) 0x7fc324000b90 con 0x7fc320040c80 2024-09-17T15:46:08.407 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:46:08.407 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-17T15:46:08.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.407+0000 7fc344d4b640 1 -- 172.21.15.64:0/59818045 >> v1:172.21.15.100:6812/4210963484 conn(0x7fc320040c80 legacy=0x7fc320043140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.407+0000 7fc344d4b640 10 monclient: shutdown 2024-09-17T15:46:08.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.407+0000 7fc344d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:08.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.407+0000 7fc344d4b640 1 -- 172.21.15.64:0/59818045 >> v1:172.21.15.64:6793/0 conn(0x7fc3400a5520 legacy=0x7fc34009fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.407+0000 7fc344d4b640 1 -- 172.21.15.64:0/59818045 shutdown_connections 2024-09-17T15:46:08.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.407+0000 7fc344d4b640 1 -- 172.21.15.64:0/59818045 >> 172.21.15.64:0/59818045 conn(0x7fc34006db30 msgr2=0x7fc3401334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:08.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.407+0000 7fc344d4b640 1 -- 172.21.15.64:0/59818045 shutdown_connections 2024-09-17T15:46:08.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.407+0000 7fc344d4b640 1 -- 172.21.15.64:0/59818045 wait complete. 2024-09-17T15:46:08.420 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-17T15:46:08.359112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163256,"kb_used_data":1060,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067784,"statfs":{"total":579820584960,"available":579653410816,"internally_reserved":0,"allocated":1085440,"data_stored":1364194,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.426027"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952807+0000","last_change":"2024-09-17T15:46:03.953098+0000","last_active":"2024-09-17T15:46:03.952807+0000","last_peered":"2024-09-17T15:46:03.952807+0000","last_clean":"2024-09-17T15:46:03.952807+0000","last_became_active":"2024-09-17T15:46:02.372737+0000","last_became_peered":"2024-09-17T15:46:02.372737+0000","last_unstale":"2024-09-17T15:46:03.952807+0000","last_undegraded":"2024-09-17T15:46:03.952807+0000","last_fullsized":"2024-09-17T15:46:03.952807+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:04:26.679936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057658000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.954285+0000","last_change":"2024-09-17T15:46:03.954609+0000","last_active":"2024-09-17T15:46:03.954285+0000","last_peered":"2024-09-17T15:46:03.954285+0000","last_clean":"2024-09-17T15:46:03.954285+0000","last_became_active":"2024-09-17T15:46:01.943967+0000","last_became_peered":"2024-09-17T15:46:01.943967+0000","last_unstale":"2024-09-17T15:46:03.954285+0000","last_undegraded":"2024-09-17T15:46:03.954285+0000","last_fullsized":"2024-09-17T15:46:03.954285+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:44:26.433116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058926900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952760+0000","last_change":"2024-09-17T15:46:03.953301+0000","last_active":"2024-09-17T15:46:03.952760+0000","last_peered":"2024-09-17T15:46:03.952760+0000","last_clean":"2024-09-17T15:46:03.952760+0000","last_became_active":"2024-09-17T15:46:01.937233+0000","last_became_peered":"2024-09-17T15:46:01.937233+0000","last_unstale":"2024-09-17T15:46:03.952760+0000","last_undegraded":"2024-09-17T15:46:03.952760+0000","last_fullsized":"2024-09-17T15:46:03.952760+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:36:15.542838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001175981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.954344+0000","last_change":"2024-09-17T15:46:03.954824+0000","last_active":"2024-09-17T15:46:03.954344+0000","last_peered":"2024-09-17T15:46:03.954344+0000","last_clean":"2024-09-17T15:46:03.954344+0000","last_became_active":"2024-09-17T15:46:01.941419+0000","last_became_peered":"2024-09-17T15:46:01.941419+0000","last_unstale":"2024-09-17T15:46:03.954344+0000","last_undegraded":"2024-09-17T15:46:03.954344+0000","last_fullsized":"2024-09-17T15:46:03.954344+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:53:39.894895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077909099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952714+0000","last_change":"2024-09-17T15:46:03.952714+0000","last_active":"2024-09-17T15:46:03.952714+0000","last_peered":"2024-09-17T15:46:03.952714+0000","last_clean":"2024-09-17T15:46:03.952714+0000","last_became_active":"2024-09-17T15:46:01.941164+0000","last_became_peered":"2024-09-17T15:46:01.941164+0000","last_unstale":"2024-09-17T15:46:03.952714+0000","last_undegraded":"2024-09-17T15:46:03.952714+0000","last_fullsized":"2024-09-17T15:46:03.952714+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:07:59.957722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051738399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.954670+0000","last_change":"2024-09-17T15:46:03.954913+0000","last_active":"2024-09-17T15:46:03.954670+0000","last_peered":"2024-09-17T15:46:03.954670+0000","last_clean":"2024-09-17T15:46:03.954670+0000","last_became_active":"2024-09-17T15:46:01.941188+0000","last_became_peered":"2024-09-17T15:46:01.941188+0000","last_unstale":"2024-09-17T15:46:03.954670+0000","last_undegraded":"2024-09-17T15:46:03.954670+0000","last_fullsized":"2024-09-17T15:46:03.954670+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:25:32.291552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087819899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.953563+0000","last_change":"2024-09-17T15:46:03.953722+0000","last_active":"2024-09-17T15:46:03.953563+0000","last_peered":"2024-09-17T15:46:03.953563+0000","last_clean":"2024-09-17T15:46:03.953563+0000","last_became_active":"2024-09-17T15:46:01.941223+0000","last_became_peered":"2024-09-17T15:46:01.941223+0000","last_unstale":"2024-09-17T15:46:03.953563+0000","last_undegraded":"2024-09-17T15:46:03.953563+0000","last_fullsized":"2024-09-17T15:46:03.953563+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:18:15.696629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012017600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952412+0000","last_change":"2024-09-17T15:46:03.952788+0000","last_active":"2024-09-17T15:46:03.952412+0000","last_peered":"2024-09-17T15:46:03.952412+0000","last_clean":"2024-09-17T15:46:03.952412+0000","last_became_active":"2024-09-17T15:46:01.937420+0000","last_became_peered":"2024-09-17T15:46:01.937420+0000","last_unstale":"2024-09-17T15:46:03.952412+0000","last_undegraded":"2024-09-17T15:46:03.952412+0000","last_fullsized":"2024-09-17T15:46:03.952412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:03:18.015826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065159300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'72","reported_seq":121,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952920+0000","last_change":"2024-09-17T15:45:56.919389+0000","last_active":"2024-09-17T15:46:03.952920+0000","last_peered":"2024-09-17T15:46:03.952920+0000","last_clean":"2024-09-17T15:46:03.952920+0000","last_became_active":"2024-09-17T15:45:56.908629+0000","last_became_peered":"2024-09-17T15:45:56.908629+0000","last_unstale":"2024-09-17T15:46:03.952920+0000","last_undegraded":"2024-09-17T15:46:03.952920+0000","last_fullsized":"2024-09-17T15:46:03.952920+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_clean_scrub_stamp":"2024-09-17T15:45:53.875287+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:44:47.696204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":1057856,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":51050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":51069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542153,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":204800,"data_stored":579978,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":51050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":51069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":579978,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:46:08.422 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:46:08.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.521+0000 7f768f0d4640 10 monclient: get_monmap_and_config 2024-09-17T15:46:08.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.521+0000 7f768f0d4640 10 monclient: build_initial_monmap 2024-09-17T15:46:08.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.521+0000 7f768f0d4640 10 monclient: monmap: 2024-09-17T15:46:08.523 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:08.523 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.523 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:08.521903+0000 2024-09-17T15:46:08.523 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:08.521903+0000 2024-09-17T15:46:08.523 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:08.523 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.523 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:08.523 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:08.523 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:08.523 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:08.523 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:08.523 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:08.524 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:08.524 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:08.524 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:08.524 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.521+0000 7f768f0d4640 1 Processor -- start 2024-09-17T15:46:08.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.521+0000 7f768f0d4640 1 -- start start 2024-09-17T15:46:08.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.521+0000 7f768f0d4640 10 monclient: init 2024-09-17T15:46:08.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f768f0d4640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:08.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f768f0d4640 10 monclient: _add_conns ranks=[7,5,6,4,0,8,3,2,1] 2024-09-17T15:46:08.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f768f0d4640 10 monclient(hunting): picked mon.noname-i con 0x7f76880a54b0 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:08.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f768f0d4640 10 monclient(hunting): picked mon.noname-h con 0x7f7688135c80 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:08.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f768f0d4640 10 monclient(hunting): picked mon.noname-d con 0x7f76881360a0 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:08.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f768f0d4640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76880aacb0 con 0x7f7688135c80 2024-09-17T15:46:08.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f768f0d4640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f768809d780 con 0x7f76881360a0 2024-09-17T15:46:08.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f768f0d4640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f768809da60 con 0x7f76880a54b0 2024-09-17T15:46:08.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f768f0d4640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:08.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f768f0d4640 10 monclient(hunting): authenticate will time out at 766.197469s 2024-09-17T15:46:08.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f768d671640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f76881360a0 0x7f7688141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:49518/0 (socket says 172.21.15.64:49518) 2024-09-17T15:46:08.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f768d671640 1 -- 172.21.15.64:0/349608447 learned_addr learned my addr 172.21.15.64:0/349608447 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:08.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f76877fe640 1 -- 172.21.15.64:0/349608447 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 321977787 0 0) 0x7f768809d780 con 0x7f76881360a0 2024-09-17T15:46:08.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f76877fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f76877fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f76877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f76877fe640 10 monclient(hunting): my global_id is 4335 2024-09-17T15:46:08.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.522+0000 7f76877fe640 1 -- 172.21.15.64:0/349608447 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7674003520 con 0x7f76881360a0 2024-09-17T15:46:08.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 1 -- 172.21.15.64:0/349608447 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1871838345 0 0) 0x7f7674003520 con 0x7f76881360a0 2024-09-17T15:46:08.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 1 -- 172.21.15.64:0/349608447 >> v1:172.21.15.100:6792/0 conn(0x7f76880a54b0 legacy=0x7f76880a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 1 -- 172.21.15.64:0/349608447 >> v1:172.21.15.100:6791/0 conn(0x7f7688135c80 legacy=0x7f768813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 1 monclient: found mon.noname-d 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 1 -- 172.21.15.64:0/349608447 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f768809dd40 con 0x7f76881360a0 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 1 -- 172.21.15.64:0/349608447 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7680001830 con 0x7f76881360a0 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f768f0d4640 5 monclient: authenticate success, global_id 4335 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f768f0d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 10 monclient: dump: 2024-09-17T15:46:08.533 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.534 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f768f0d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 1 -- 172.21.15.64:0/349608447 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f76800019d0 con 0x7f76881360a0 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 1 -- 172.21.15.64:0/349608447 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7680004be0 con 0x7f76881360a0 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f76877fe640 10 monclient: dump: 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.535 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f768f0d4640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f768f0d4640 4 set_mon_vals no callback set 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f768f0d4640 10 monclient: shutdown 2024-09-17T15:46:08.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f768f0d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f768f0d4640 1 -- 172.21.15.64:0/349608447 >> v1:172.21.15.64:6792/0 conn(0x7f76881360a0 legacy=0x7f7688141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f768f0d4640 1 -- 172.21.15.64:0/349608447 shutdown_connections 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f768f0d4640 1 -- 172.21.15.64:0/349608447 >> 172.21.15.64:0/349608447 conn(0x7f768806daa0 msgr2=0x7f76880ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f768f0d4640 1 -- 172.21.15.64:0/349608447 shutdown_connections 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.523+0000 7f768f0d4640 1 -- 172.21.15.64:0/349608447 wait complete. 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.524+0000 7f768f0d4640 10 monclient: build_initial_monmap 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.524+0000 7f768f0d4640 10 monclient: monmap: 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:08.537 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.524+0000 7f768f0d4640 1 Processor -- start 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.524+0000 7f768f0d4640 1 -- start start 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.524+0000 7f768f0d4640 10 monclient: init 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f768f0d4640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:08.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f768f0d4640 10 monclient: _add_conns ranks=[2,1,4,3,7,5,8,0,6] 2024-09-17T15:46:08.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f768f0d4640 10 monclient(hunting): picked mon.noname-b con 0x7f76880a54b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:08.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f768f0d4640 10 monclient(hunting): picked mon.noname-f con 0x7f7688135c80 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:08.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f768f0d4640 10 monclient(hunting): picked mon.noname-c con 0x7f76881360a0 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:08.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f768f0d4640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76880a6de0 con 0x7f7688135c80 2024-09-17T15:46:08.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f768f0d4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76880a25b0 con 0x7f76880a54b0 2024-09-17T15:46:08.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f768f0d4640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76881ce180 con 0x7f76881360a0 2024-09-17T15:46:08.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f768f0d4640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:08.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f768f0d4640 10 monclient(hunting): authenticate will time out at 766.200404s 2024-09-17T15:46:08.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f768ce70640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f76880a54b0 0x7f76881cd360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46862/0 (socket says 172.21.15.64:46862) 2024-09-17T15:46:08.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f768ce70640 1 -- 172.21.15.64:0/3086132213 learned_addr learned my addr 172.21.15.64:0/3086132213 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:08.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1363380691 0 0) 0x7f76881ce180 con 0x7f76881360a0 2024-09-17T15:46:08.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 10 monclient(hunting): my global_id is 4459 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7668003520 con 0x7f76881360a0 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4259820353 0 0) 0x7f76880a25b0 con 0x7f76880a54b0 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 10 monclient(hunting): my global_id is 4394 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76881ce180 con 0x7f76880a54b0 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3297652441 0 0) 0x7f7670001270 con 0x7f7688135c80 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 10 monclient(hunting): my global_id is 4420 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76880a25b0 con 0x7f7688135c80 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2161372314 0 0) 0x7f7668003520 con 0x7f76881360a0 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7670001270 con 0x7f76881360a0 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.525+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f76800016f0 con 0x7f76881360a0 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient(hunting): dump: 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.541 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 495104514 0 0) 0x7f76881ce180 con 0x7f76880a54b0 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7668003520 con 0x7f76880a54b0 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7678001e90 con 0x7f76880a54b0 2024-09-17T15:46:08.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:08.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:08.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient(hunting): dump: 2024-09-17T15:46:08.543 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.543 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.543 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.543 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.543 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.543 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.543 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.543 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.543 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.543 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.544 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.544 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.544 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.544 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.544 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.544 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1352113105 0 0) 0x7f7670001270 con 0x7f76881360a0 2024-09-17T15:46:08.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 >> v1:172.21.15.64:6790/0 conn(0x7f76880a54b0 legacy=0x7f76881cd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 >> v1:172.21.15.100:6789/0 conn(0x7f7688135c80 legacy=0x7f76881cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient: _finish_hunting 0 2024-09-17T15:46:08.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 1 monclient: found mon.c 2024-09-17T15:46:08.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:08.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7688135270 con 0x7f76881360a0 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient: _finish_auth 0 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient: _check_auth_tickets 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f768f0d4640 5 monclient: authenticate success, global_id 4459 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f768f0d4640 10 monclient: _renew_subs 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f768f0d4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f768f0d4640 1 -- 172.21.15.64:0/3086132213 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f76880a0250 con 0x7f76881360a0 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f768f0d4640 10 monclient: _renew_subs 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f768f0d4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f768f0d4640 1 -- 172.21.15.64:0/3086132213 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f76880a0db0 con 0x7f76881360a0 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f76800019b0 con 0x7f76881360a0 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:08.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7680004dd0 con 0x7f76881360a0 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f7686ffd640 4 set_mon_vals no callback set 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.526+0000 7f76857fa640 10 monclient: dump: 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.546 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.547 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.547 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.547 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.547 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.547 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.547 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.527+0000 7f768f0d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:08.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.527+0000 7f768f0d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:08.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.527+0000 7f768f0d4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:08.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.527+0000 7f768f0d4640 1 -- 172.21.15.64:0/3086132213 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f766c005080 con 0x7f76881360a0 2024-09-17T15:46:08.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.527+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f76800116b0 con 0x7f76881360a0 2024-09-17T15:46:08.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.527+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1513317904 0 0) 0x7f768004e800 con 0x7f76881360a0 2024-09-17T15:46:08.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.528+0000 7f768ce70640 10 monclient: get_auth_request con 0x7f76680408b0 auth_method 0 2024-09-17T15:46:08.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.532+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f7680017640 con 0x7f76881360a0 2024-09-17T15:46:08.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.532+0000 7f76857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:08.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.532+0000 7f76857fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:08.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.714+0000 7f768f0d4640 1 -- 172.21.15.64:0/3086132213 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f766c000b90 con 0x7f76680408b0 2024-09-17T15:46:08.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.715+0000 7f76857fa640 1 -- 172.21.15.64:0/3086132213 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34189 (unknown 2965378022 0 1866897357) 0x7f766c000b90 con 0x7f76680408b0 2024-09-17T15:46:08.717 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:46:08.718 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-17T15:46:08.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.718+0000 7f768f0d4640 1 -- 172.21.15.64:0/3086132213 >> v1:172.21.15.100:6812/4210963484 conn(0x7f76680408b0 legacy=0x7f7668042d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.718+0000 7f768f0d4640 10 monclient: shutdown 2024-09-17T15:46:08.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.718+0000 7f768f0d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:08.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.718+0000 7f768f0d4640 1 -- 172.21.15.64:0/3086132213 >> v1:172.21.15.64:6791/0 conn(0x7f76881360a0 legacy=0x7f76881d5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.718+0000 7f768f0d4640 1 -- 172.21.15.64:0/3086132213 shutdown_connections 2024-09-17T15:46:08.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.718+0000 7f768f0d4640 1 -- 172.21.15.64:0/3086132213 >> 172.21.15.64:0/3086132213 conn(0x7f768806daa0 msgr2=0x7f76880a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:08.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.718+0000 7f768f0d4640 1 -- 172.21.15.64:0/3086132213 shutdown_connections 2024-09-17T15:46:08.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.718+0000 7f768f0d4640 1 -- 172.21.15.64:0/3086132213 wait complete. 2024-09-17T15:46:08.730 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-17T15:46:08.359112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163256,"kb_used_data":1060,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067784,"statfs":{"total":579820584960,"available":579653410816,"internally_reserved":0,"allocated":1085440,"data_stored":1364194,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.426027"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952807+0000","last_change":"2024-09-17T15:46:03.953098+0000","last_active":"2024-09-17T15:46:03.952807+0000","last_peered":"2024-09-17T15:46:03.952807+0000","last_clean":"2024-09-17T15:46:03.952807+0000","last_became_active":"2024-09-17T15:46:02.372737+0000","last_became_peered":"2024-09-17T15:46:02.372737+0000","last_unstale":"2024-09-17T15:46:03.952807+0000","last_undegraded":"2024-09-17T15:46:03.952807+0000","last_fullsized":"2024-09-17T15:46:03.952807+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:04:26.679936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057658000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.954285+0000","last_change":"2024-09-17T15:46:03.954609+0000","last_active":"2024-09-17T15:46:03.954285+0000","last_peered":"2024-09-17T15:46:03.954285+0000","last_clean":"2024-09-17T15:46:03.954285+0000","last_became_active":"2024-09-17T15:46:01.943967+0000","last_became_peered":"2024-09-17T15:46:01.943967+0000","last_unstale":"2024-09-17T15:46:03.954285+0000","last_undegraded":"2024-09-17T15:46:03.954285+0000","last_fullsized":"2024-09-17T15:46:03.954285+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:44:26.433116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058926900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952760+0000","last_change":"2024-09-17T15:46:03.953301+0000","last_active":"2024-09-17T15:46:03.952760+0000","last_peered":"2024-09-17T15:46:03.952760+0000","last_clean":"2024-09-17T15:46:03.952760+0000","last_became_active":"2024-09-17T15:46:01.937233+0000","last_became_peered":"2024-09-17T15:46:01.937233+0000","last_unstale":"2024-09-17T15:46:03.952760+0000","last_undegraded":"2024-09-17T15:46:03.952760+0000","last_fullsized":"2024-09-17T15:46:03.952760+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:36:15.542838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001175981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.954344+0000","last_change":"2024-09-17T15:46:03.954824+0000","last_active":"2024-09-17T15:46:03.954344+0000","last_peered":"2024-09-17T15:46:03.954344+0000","last_clean":"2024-09-17T15:46:03.954344+0000","last_became_active":"2024-09-17T15:46:01.941419+0000","last_became_peered":"2024-09-17T15:46:01.941419+0000","last_unstale":"2024-09-17T15:46:03.954344+0000","last_undegraded":"2024-09-17T15:46:03.954344+0000","last_fullsized":"2024-09-17T15:46:03.954344+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:53:39.894895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077909099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952714+0000","last_change":"2024-09-17T15:46:03.952714+0000","last_active":"2024-09-17T15:46:03.952714+0000","last_peered":"2024-09-17T15:46:03.952714+0000","last_clean":"2024-09-17T15:46:03.952714+0000","last_became_active":"2024-09-17T15:46:01.941164+0000","last_became_peered":"2024-09-17T15:46:01.941164+0000","last_unstale":"2024-09-17T15:46:03.952714+0000","last_undegraded":"2024-09-17T15:46:03.952714+0000","last_fullsized":"2024-09-17T15:46:03.952714+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:07:59.957722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051738399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.954670+0000","last_change":"2024-09-17T15:46:03.954913+0000","last_active":"2024-09-17T15:46:03.954670+0000","last_peered":"2024-09-17T15:46:03.954670+0000","last_clean":"2024-09-17T15:46:03.954670+0000","last_became_active":"2024-09-17T15:46:01.941188+0000","last_became_peered":"2024-09-17T15:46:01.941188+0000","last_unstale":"2024-09-17T15:46:03.954670+0000","last_undegraded":"2024-09-17T15:46:03.954670+0000","last_fullsized":"2024-09-17T15:46:03.954670+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:25:32.291552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087819899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.953563+0000","last_change":"2024-09-17T15:46:03.953722+0000","last_active":"2024-09-17T15:46:03.953563+0000","last_peered":"2024-09-17T15:46:03.953563+0000","last_clean":"2024-09-17T15:46:03.953563+0000","last_became_active":"2024-09-17T15:46:01.941223+0000","last_became_peered":"2024-09-17T15:46:01.941223+0000","last_unstale":"2024-09-17T15:46:03.953563+0000","last_undegraded":"2024-09-17T15:46:03.953563+0000","last_fullsized":"2024-09-17T15:46:03.953563+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:18:15.696629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012017600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952412+0000","last_change":"2024-09-17T15:46:03.952788+0000","last_active":"2024-09-17T15:46:03.952412+0000","last_peered":"2024-09-17T15:46:03.952412+0000","last_clean":"2024-09-17T15:46:03.952412+0000","last_became_active":"2024-09-17T15:46:01.937420+0000","last_became_peered":"2024-09-17T15:46:01.937420+0000","last_unstale":"2024-09-17T15:46:03.952412+0000","last_undegraded":"2024-09-17T15:46:03.952412+0000","last_fullsized":"2024-09-17T15:46:03.952412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:03:18.015826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065159300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'72","reported_seq":121,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-17T15:46:03.952920+0000","last_change":"2024-09-17T15:45:56.919389+0000","last_active":"2024-09-17T15:46:03.952920+0000","last_peered":"2024-09-17T15:46:03.952920+0000","last_clean":"2024-09-17T15:46:03.952920+0000","last_became_active":"2024-09-17T15:45:56.908629+0000","last_became_peered":"2024-09-17T15:45:56.908629+0000","last_unstale":"2024-09-17T15:46:03.952920+0000","last_undegraded":"2024-09-17T15:46:03.952920+0000","last_fullsized":"2024-09-17T15:46:03.952920+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_clean_scrub_stamp":"2024-09-17T15:45:53.875287+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:44:47.696204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":1057856,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":51050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":51069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542153,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":204800,"data_stored":579978,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":51050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":51069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":579978,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:46:08.731 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-17T15:46:08.731 DEBUG:tasks.mon_thrash:Yielding 2024-09-17T15:46:08.731 INFO:teuthology.run_tasks:Running task exec... 2024-09-17T15:46:08.741 INFO:teuthology.task.exec:Executing custom commands... 2024-09-17T15:46:08.741 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T15:46:08.741 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:46:08.741 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-17T15:46:08.741 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726587968, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-17T15:46:08.742 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-17T15:46:08.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:46:08.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.840+0000 7f20dc44f640 10 monclient: get_monmap_and_config 2024-09-17T15:46:08.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.840+0000 7f20dc44f640 10 monclient: build_initial_monmap 2024-09-17T15:46:08.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.840+0000 7f20dc44f640 10 monclient: monmap: 2024-09-17T15:46:08.842 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:08.842 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.842 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:08.840920+0000 2024-09-17T15:46:08.842 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:08.840920+0000 2024-09-17T15:46:08.842 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:08.842 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.842 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:08.842 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:08.843 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.840+0000 7f20dc44f640 1 Processor -- start 2024-09-17T15:46:08.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.840+0000 7f20dc44f640 1 -- start start 2024-09-17T15:46:08.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.840+0000 7f20dc44f640 10 monclient: init 2024-09-17T15:46:08.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20dc44f640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:08.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20dc44f640 10 monclient: _add_conns ranks=[2,8,6,0,7,4,5,1,3] 2024-09-17T15:46:08.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20dc44f640 10 monclient(hunting): picked mon.noname-b con 0x7f20d40a54b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:08.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20dc44f640 10 monclient(hunting): picked mon.noname-e con 0x7f20d4135c80 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:08.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20dc44f640 10 monclient(hunting): picked mon.noname-d con 0x7f20d41360a0 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:08.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20dc44f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d40aacb0 con 0x7f20d40a54b0 2024-09-17T15:46:08.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20dc44f640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d409d780 con 0x7f20d41360a0 2024-09-17T15:46:08.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20dc44f640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d409da60 con 0x7f20d4135c80 2024-09-17T15:46:08.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20dc44f640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:08.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20dc44f640 10 monclient(hunting): authenticate will time out at 766.516449s 2024-09-17T15:46:08.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d99ea640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f20d4135c80 0x7f20d413df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38692/0 (socket says 172.21.15.64:38692) 2024-09-17T15:46:08.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d99ea640 1 -- 172.21.15.64:0/912143255 learned_addr learned my addr 172.21.15.64:0/912143255 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:08.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20da1eb640 1 --1- 172.21.15.64:0/912143255 >> v1:172.21.15.64:6790/0 conn(0x7f20d40a54b0 0x7f20d40a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46876/0 (socket says 172.21.15.64:46876) 2024-09-17T15:46:08.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 1 -- 172.21.15.64:0/912143255 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 131495876 0 0) 0x7f20d409d780 con 0x7f20d41360a0 2024-09-17T15:46:08.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 10 monclient(hunting): my global_id is 4353 2024-09-17T15:46:08.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 1 -- 172.21.15.64:0/912143255 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20c0003520 con 0x7f20d41360a0 2024-09-17T15:46:08.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 1 -- 172.21.15.64:0/912143255 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2981236031 0 0) 0x7f20d40aacb0 con 0x7f20d40a54b0 2024-09-17T15:46:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 10 monclient(hunting): my global_id is 4403 2024-09-17T15:46:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 1 -- 172.21.15.64:0/912143255 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20d409d780 con 0x7f20d40a54b0 2024-09-17T15:46:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 1 -- 172.21.15.64:0/912143255 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2100945255 0 0) 0x7f20d409da60 con 0x7f20d4135c80 2024-09-17T15:46:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 10 monclient(hunting): my global_id is 4391 2024-09-17T15:46:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.841+0000 7f20d91e9640 1 -- 172.21.15.64:0/912143255 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20d40aacb0 con 0x7f20d4135c80 2024-09-17T15:46:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 1 -- 172.21.15.64:0/912143255 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2402781886 0 0) 0x7f20d409d780 con 0x7f20d40a54b0 2024-09-17T15:46:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 1 -- 172.21.15.64:0/912143255 >> v1:172.21.15.64:6793/0 conn(0x7f20d4135c80 legacy=0x7f20d413df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 1 -- 172.21.15.64:0/912143255 >> v1:172.21.15.64:6792/0 conn(0x7f20d41360a0 legacy=0x7f20d4141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 10 monclient: _finish_hunting 0 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 1 monclient: found mon.noname-b 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 1 -- 172.21.15.64:0/912143255 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d409dd40 con 0x7f20d40a54b0 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 10 monclient: _finish_auth 0 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 10 monclient: _check_auth_tickets 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 1 -- 172.21.15.64:0/912143255 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f20d0001810 con 0x7f20d40a54b0 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 10 monclient: dump: 2024-09-17T15:46:08.855 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.856 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20dc44f640 5 monclient: authenticate success, global_id 4403 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20dc44f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 1 -- 172.21.15.64:0/912143255 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20d0001a30 con 0x7f20d40a54b0 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 1 -- 172.21.15.64:0/912143255 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f20d0001ca0 con 0x7f20d40a54b0 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20dc44f640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20dc44f640 4 set_mon_vals no callback set 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20d91e9640 10 monclient: dump: 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.857 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.858 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.858 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.858 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.858 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.858 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.858 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.858 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.858 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.858 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.858 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.858 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.858 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.858 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20dc44f640 10 monclient: shutdown 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20dc44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20dc44f640 1 -- 172.21.15.64:0/912143255 >> v1:172.21.15.64:6790/0 conn(0x7f20d40a54b0 legacy=0x7f20d40a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20dc44f640 1 -- 172.21.15.64:0/912143255 shutdown_connections 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.842+0000 7f20dc44f640 1 -- 172.21.15.64:0/912143255 >> 172.21.15.64:0/912143255 conn(0x7f20d406daa0 msgr2=0x7f20d40ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.843+0000 7f20dc44f640 1 -- 172.21.15.64:0/912143255 shutdown_connections 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.843+0000 7f20dc44f640 1 -- 172.21.15.64:0/912143255 wait complete. 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.843+0000 7f20dc44f640 10 monclient: build_initial_monmap 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.843+0000 7f20dc44f640 10 monclient: monmap: 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:08.859 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:08.860 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:08.860 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:08.860 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:08.860 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:08.860 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:08.860 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:08.860 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:08.860 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:08.860 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:08.860 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:08.860 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:08.860 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.843+0000 7f20dc44f640 1 Processor -- start 2024-09-17T15:46:08.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.843+0000 7f20dc44f640 1 -- start start 2024-09-17T15:46:08.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.843+0000 7f20dc44f640 10 monclient: init 2024-09-17T15:46:08.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20dc44f640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:08.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20dc44f640 10 monclient: _add_conns ranks=[8,4,5,3,6,7,2,0,1] 2024-09-17T15:46:08.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20dc44f640 10 monclient(hunting): picked mon.noname-e con 0x7f20d40a54b0 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:08.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20dc44f640 10 monclient(hunting): picked mon.noname-c con 0x7f20d4135c80 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:08.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20dc44f640 10 monclient(hunting): picked mon.noname-h con 0x7f20d41360a0 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:08.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20dc44f640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d40a0a50 con 0x7f20d4135c80 2024-09-17T15:46:08.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20dc44f640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d409f120 con 0x7f20d41360a0 2024-09-17T15:46:08.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20dc44f640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d40a6c30 con 0x7f20d40a54b0 2024-09-17T15:46:08.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20da1eb640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f20d40a54b0 0x7f20d409e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38702/0 (socket says 172.21.15.64:38702) 2024-09-17T15:46:08.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20dc44f640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:08.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20da1eb640 1 -- 172.21.15.64:0/1226407964 learned_addr learned my addr 172.21.15.64:0/1226407964 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:08.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20dc44f640 10 monclient(hunting): authenticate will time out at 766.519582s 2024-09-17T15:46:08.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3418950533 0 0) 0x7f20d40a6c30 con 0x7f20d40a54b0 2024-09-17T15:46:08.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 10 monclient(hunting): my global_id is 4400 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20b8003520 con 0x7f20d40a54b0 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 955777803 0 0) 0x7f20d40a0a50 con 0x7f20d4135c80 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 10 monclient(hunting): my global_id is 4495 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20d40a6c30 con 0x7f20d4135c80 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1762468032 0 0) 0x7f20d409f120 con 0x7f20d41360a0 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.844+0000 7f20c6ffd640 10 monclient(hunting): my global_id is 4379 2024-09-17T15:46:08.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20d40a0a50 con 0x7f20d41360a0 2024-09-17T15:46:08.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3702569031 0 0) 0x7f20b8003520 con 0x7f20d40a54b0 2024-09-17T15:46:08.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20d409f120 con 0x7f20d40a54b0 2024-09-17T15:46:08.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f20d0001c00 con 0x7f20d40a54b0 2024-09-17T15:46:08.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:08.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:46:08.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient(hunting): dump: 2024-09-17T15:46:08.864 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.864 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.864 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.864 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.864 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.864 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.864 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1032721070 0 0) 0x7f20d40a6c30 con 0x7f20d4135c80 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20b8003520 con 0x7f20d4135c80 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f20c80016e0 con 0x7f20d4135c80 2024-09-17T15:46:08.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:08.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:08.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient(hunting): dump: 2024-09-17T15:46:08.866 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.866 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.866 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.866 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.866 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.866 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.866 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.866 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.866 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.866 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.867 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.867 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.867 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.867 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.867 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.867 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 283876405 0 0) 0x7f20d40a0a50 con 0x7f20d41360a0 2024-09-17T15:46:08.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20d40a6c30 con 0x7f20d41360a0 2024-09-17T15:46:08.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f20cc001800 con 0x7f20d41360a0 2024-09-17T15:46:08.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:08.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:46:08.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient(hunting): dump: 2024-09-17T15:46:08.868 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.868 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.868 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.868 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.869 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.870 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.870 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 725029503 0 0) 0x7f20d409f120 con 0x7f20d40a54b0 2024-09-17T15:46:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 >> v1:172.21.15.100:6791/0 conn(0x7f20d41360a0 legacy=0x7f20d41d62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 >> v1:172.21.15.64:6791/0 conn(0x7f20d4135c80 legacy=0x7f20d409ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 monclient: found mon.e 2024-09-17T15:46:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:08.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:08.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d4132840 con 0x7f20d40a54b0 2024-09-17T15:46:08.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:46:08.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:08.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:08.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20dc44f640 5 monclient: authenticate success, global_id 4400 2024-09-17T15:46:08.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20dc44f640 10 monclient: _renew_subs 2024-09-17T15:46:08.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20dc44f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:08.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20dc44f640 1 -- 172.21.15.64:0/1226407964 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f20d40a0d30 con 0x7f20d40a54b0 2024-09-17T15:46:08.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20dc44f640 10 monclient: _renew_subs 2024-09-17T15:46:08.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20dc44f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:08.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20dc44f640 1 -- 172.21.15.64:0/1226407964 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f20d409f530 con 0x7f20d40a54b0 2024-09-17T15:46:08.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20d0004e70 con 0x7f20d40a54b0 2024-09-17T15:46:08.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:08.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f20d00018f0 con 0x7f20d40a54b0 2024-09-17T15:46:08.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:08.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20d89e8640 4 set_mon_vals no callback set 2024-09-17T15:46:08.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:08.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.845+0000 7f20c6ffd640 10 monclient: dump: 2024-09-17T15:46:08.874 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:08.874 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:08.874 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.874 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:08.875 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:08.875 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:08.875 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:08.875 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:08.875 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:08.875 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:08.875 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:08.876 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:08.876 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:08.876 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:08.876 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:08.876 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:08.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.846+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f20d0011aa0 con 0x7f20d40a54b0 2024-09-17T15:46:08.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.846+0000 7f20dc44f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:08.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.846+0000 7f20dc44f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:08.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.846+0000 7f20dc44f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:08.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.846+0000 7f20dc44f640 1 -- 172.21.15.64:0/1226407964 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20bc005080 con 0x7f20d40a54b0 2024-09-17T15:46:08.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.846+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1513317904 0 0) 0x7f20d004e9c0 con 0x7f20d40a54b0 2024-09-17T15:46:08.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.846+0000 7f20d99ea640 10 monclient: get_auth_request con 0x7f20b8040bf0 auth_method 0 2024-09-17T15:46:08.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.851+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.8 v1:172.21.15.64:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f20d00177a0 con 0x7f20d40a54b0 2024-09-17T15:46:08.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.851+0000 7f20c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:08.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:08.851+0000 7f20c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:09.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.113+0000 7f20dc44f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:46:09.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.113+0000 7f20dc44f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:09.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.113+0000 7f20dc44f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:09.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.113+0000 7f20dc44f640 1 -- 172.21.15.64:0/1226407964 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f20bc005370 con 0x7f20d40a54b0 2024-09-17T15:46:09.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.114+0000 7f20c6ffd640 1 -- 172.21.15.64:0/1226407964 <== mon.8 v1:172.21.15.64:6793/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 3330327295) 0x7f20d0020a00 con 0x7f20d40a54b0 2024-09-17T15:46:09.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.114+0000 7f20c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:09.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.114+0000 7f20c6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:09.116 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":8,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:46:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.116+0000 7f20dc44f640 1 -- 172.21.15.64:0/1226407964 >> v1:172.21.15.100:6812/4210963484 conn(0x7f20b8040bf0 legacy=0x7f20b80430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.116+0000 7f20dc44f640 10 monclient: shutdown 2024-09-17T15:46:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.116+0000 7f20dc44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.116+0000 7f20dc44f640 1 -- 172.21.15.64:0/1226407964 >> v1:172.21.15.64:6793/0 conn(0x7f20d40a54b0 legacy=0x7f20d409e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.116+0000 7f20dc44f640 1 -- 172.21.15.64:0/1226407964 shutdown_connections 2024-09-17T15:46:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.116+0000 7f20dc44f640 1 -- 172.21.15.64:0/1226407964 >> 172.21.15.64:0/1226407964 conn(0x7f20d406daa0 msgr2=0x7f20d40a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.116+0000 7f20dc44f640 1 -- 172.21.15.64:0/1226407964 shutdown_connections 2024-09-17T15:46:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.116+0000 7f20dc44f640 1 -- 172.21.15.64:0/1226407964 wait complete. 2024-09-17T15:46:09.128 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-17T15:46:09.128 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:46:09.128 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-17T15:46:09.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:46:09.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.229+0000 7f5bd6239640 10 monclient: get_monmap_and_config 2024-09-17T15:46:09.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.229+0000 7f5bd6239640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.229+0000 7f5bd6239640 10 monclient: monmap: 2024-09-17T15:46:09.231 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:09.231 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.231 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:09.230592+0000 2024-09-17T15:46:09.232 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:09.230592+0000 2024-09-17T15:46:09.232 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.232 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.232 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:09.232 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:09.232 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:09.232 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:09.232 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:09.232 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:09.232 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:09.232 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:09.232 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:09.232 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.230+0000 7f5bd6239640 1 Processor -- start 2024-09-17T15:46:09.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.230+0000 7f5bd6239640 1 -- start start 2024-09-17T15:46:09.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.230+0000 7f5bd6239640 10 monclient: init 2024-09-17T15:46:09.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.230+0000 7f5bd6239640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 10 monclient: _add_conns ranks=[6,1,3,2,7,4,5,8,0] 2024-09-17T15:46:09.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 10 monclient(hunting): picked mon.noname-d con 0x7f5bd00a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 10 monclient(hunting): picked mon.noname-f con 0x7f5bd0135b50 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 10 monclient(hunting): picked mon.noname-g con 0x7f5bd0135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bd009d6f0 con 0x7f5bd0135b50 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bd009d9d0 con 0x7f5bd0135f70 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bd009dcb0 con 0x7f5bd00a5520 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 10 monclient(hunting): authenticate will time out at 766.906090s 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bcf7fe640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f5bd00a5520 0x7f5bd00a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:49534/0 (socket says 172.21.15.64:49534) 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bcf7fe640 1 -- 172.21.15.64:0/1856531853 learned_addr learned my addr 172.21.15.64:0/1856531853 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 1 -- 172.21.15.64:0/1856531853 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 390933734 0 0) 0x7f5bd009dcb0 con 0x7f5bd00a5520 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient(hunting): my global_id is 4362 2024-09-17T15:46:09.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 1 -- 172.21.15.64:0/1856531853 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bb8003520 con 0x7f5bd00a5520 2024-09-17T15:46:09.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 1 -- 172.21.15.64:0/1856531853 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4072377184 0 0) 0x7f5bb8003520 con 0x7f5bd00a5520 2024-09-17T15:46:09.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 1 -- 172.21.15.64:0/1856531853 >> v1:172.21.15.100:6790/0 conn(0x7f5bd0135f70 legacy=0x7f5bd01415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 1 -- 172.21.15.64:0/1856531853 >> v1:172.21.15.100:6789/0 conn(0x7f5bd0135b50 legacy=0x7f5bd013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 1 monclient: found mon.noname-d 2024-09-17T15:46:09.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:46:09.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 1 -- 172.21.15.64:0/1856531853 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd009ccd0 con 0x7f5bd00a5520 2024-09-17T15:46:09.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 1 -- 172.21.15.64:0/1856531853 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5bc0001830 con 0x7f5bd00a5520 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 5 monclient: authenticate success, global_id 4362 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient: dump: 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.242 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 1 -- 172.21.15.64:0/1856531853 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5bc0001a50 con 0x7f5bd00a5520 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 1 -- 172.21.15.64:0/1856531853 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5bc0004be0 con 0x7f5bd00a5520 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bd6239640 4 set_mon_vals no callback set 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.231+0000 7f5bce7fc640 10 monclient: dump: 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.232+0000 7f5bd6239640 10 monclient: shutdown 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.232+0000 7f5bd6239640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.232+0000 7f5bd6239640 1 -- 172.21.15.64:0/1856531853 >> v1:172.21.15.64:6792/0 conn(0x7f5bd00a5520 legacy=0x7f5bd00a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.232+0000 7f5bd6239640 1 -- 172.21.15.64:0/1856531853 shutdown_connections 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.232+0000 7f5bd6239640 1 -- 172.21.15.64:0/1856531853 >> 172.21.15.64:0/1856531853 conn(0x7f5bd006db30 msgr2=0x7f5bd00ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.232+0000 7f5bd6239640 1 -- 172.21.15.64:0/1856531853 shutdown_connections 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.232+0000 7f5bd6239640 1 -- 172.21.15.64:0/1856531853 wait complete. 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.232+0000 7f5bd6239640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.232+0000 7f5bd6239640 10 monclient: monmap: 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:09.245 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bd6239640 1 Processor -- start 2024-09-17T15:46:09.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bd6239640 1 -- start start 2024-09-17T15:46:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bd6239640 10 monclient: init 2024-09-17T15:46:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bd6239640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bd6239640 10 monclient: _add_conns ranks=[1,6,5,8,3,0,2,7,4] 2024-09-17T15:46:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bd6239640 10 monclient(hunting): picked mon.noname-f con 0x7f5bd00a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bd6239640 10 monclient(hunting): picked mon.noname-d con 0x7f5bd0135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bd6239640 10 monclient(hunting): picked mon.noname-h con 0x7f5bd0135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bd6239640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bd01388e0 con 0x7f5bd00a5520 2024-09-17T15:46:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bd6239640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bd00a6420 con 0x7f5bd0135f70 2024-09-17T15:46:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bd6239640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bd013ac40 con 0x7f5bd0135b50 2024-09-17T15:46:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bd6239640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bd6239640 10 monclient(hunting): authenticate will time out at 766.908850s 2024-09-17T15:46:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bceffd640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f5bd0135b50 0x7f5bd0137580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:49540/0 (socket says 172.21.15.64:49540) 2024-09-17T15:46:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.233+0000 7f5bceffd640 1 -- 172.21.15.64:0/2497165021 learned_addr learned my addr 172.21.15.64:0/2497165021 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:09.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1787116928 0 0) 0x7f5bd013ac40 con 0x7f5bd0135b50 2024-09-17T15:46:09.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): my global_id is 4371 2024-09-17T15:46:09.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bbc003520 con 0x7f5bd0135b50 2024-09-17T15:46:09.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 951394712 0 0) 0x7f5bd01388e0 con 0x7f5bd00a5520 2024-09-17T15:46:09.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): my global_id is 4465 2024-09-17T15:46:09.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bd013ac40 con 0x7f5bd00a5520 2024-09-17T15:46:09.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1968487149 0 0) 0x7f5bd00a6420 con 0x7f5bd0135f70 2024-09-17T15:46:09.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): my global_id is 4388 2024-09-17T15:46:09.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bd01388e0 con 0x7f5bd0135f70 2024-09-17T15:46:09.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3823516819 0 0) 0x7f5bbc003520 con 0x7f5bd0135b50 2024-09-17T15:46:09.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5bd00a6420 con 0x7f5bd0135b50 2024-09-17T15:46:09.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5bc4001b90 con 0x7f5bd0135b50 2024-09-17T15:46:09.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:09.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:46:09.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): dump: 2024-09-17T15:46:09.249 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.249 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2148052939 0 0) 0x7f5bd00a6420 con 0x7f5bd0135b50 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 >> v1:172.21.15.100:6791/0 conn(0x7f5bd0135f70 legacy=0x7f5bd0137c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 >> v1:172.21.15.100:6789/0 conn(0x7f5bd00a5520 legacy=0x7f5bd0136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 monclient: found mon.d 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd013a3a0 con 0x7f5bd0135b50 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bd6239640 5 monclient: authenticate success, global_id 4371 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bd6239640 10 monclient: _renew_subs 2024-09-17T15:46:09.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bd6239640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bd6239640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd0139440 con 0x7f5bd0135b50 2024-09-17T15:46:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bd6239640 10 monclient: _renew_subs 2024-09-17T15:46:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bd6239640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bd6239640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f5bd01d7160 con 0x7f5bd0135b50 2024-09-17T15:46:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5bc40038c0 con 0x7f5bd0135b50 2024-09-17T15:46:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5bc4003df0 con 0x7f5bd0135b50 2024-09-17T15:46:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bcdffb640 4 set_mon_vals no callback set 2024-09-17T15:46:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.234+0000 7f5bb7fff640 10 monclient: dump: 2024-09-17T15:46:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.235+0000 7f5bd6239640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.235+0000 7f5bd6239640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5bb00065c0 con 0x7f5bb0002860 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.235+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f5bc4003f90 con 0x7f5bd0135b50 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.235+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1513317904 0 0) 0x7f5bc404ebb0 con 0x7f5bd0135b50 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1600893006 0 0) 0x7f5bb00065c0 con 0x7f5bb0002860 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 10 monclient(hunting): my global_id is 4356 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bd00a6420 con 0x7f5bb0002860 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bcffff640 10 monclient: get_auth_request con 0x7f5bbc040530 auth_method 0 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1728206760 0 0) 0x7f5bd00a6420 con 0x7f5bb0002860 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5bb00065c0 con 0x7f5bb0002860 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5bc0004be0 con 0x7f5bb0002860 2024-09-17T15:46:09.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 10 monclient: dump: 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.256 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.256 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.256 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.256 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 212312793 0 0) 0x7f5bb00065c0 con 0x7f5bb0002860 2024-09-17T15:46:09.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.64:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5bb000a160 con 0x7f5bb0002860 2024-09-17T15:46:09.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5bc00015b0 con 0x7f5bb0002860 2024-09-17T15:46:09.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:09.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 >> v1:172.21.15.64:6789/0 conn(0x7f5bb0002860 legacy=0x7f5bb0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.236+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 >> v1:172.21.15.64:6789/0 conn(0x7f5bb0002860 legacy=0x7f5bb0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.242+0000 7f5bd6239640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:09.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bd6239640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5bb0009e70 con 0x7f5bb0009130 2024-09-17T15:46:09.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 942451033 0 0) 0x7f5bb0009e70 con 0x7f5bb0009130 2024-09-17T15:46:09.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 10 monclient(hunting): my global_id is 4365 2024-09-17T15:46:09.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bb00065c0 con 0x7f5bb0009130 2024-09-17T15:46:09.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1647365819 0 0) 0x7f5bb00065c0 con 0x7f5bb0009130 2024-09-17T15:46:09.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5bb0009e70 con 0x7f5bb0009130 2024-09-17T15:46:09.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5bc0005000 con 0x7f5bb0009130 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 10 monclient: dump: 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.259 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.260 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.260 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.260 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.260 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 304999315 0 0) 0x7f5bb0009e70 con 0x7f5bb0009130 2024-09-17T15:46:09.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 --> v1:172.21.15.64:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5bb000c410 con 0x7f5bb0009130 2024-09-17T15:46:09.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 4136099086) 0x7f5bc00051b0 con 0x7f5bb0009130 2024-09-17T15:46:09.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:09.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:09.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 >> v1:172.21.15.64:6789/0 conn(0x7f5bb0009130 legacy=0x7f5bb00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.243+0000 7f5bb7fff640 1 -- 172.21.15.64:0/2497165021 >> v1:172.21.15.64:6789/0 conn(0x7f5bb0009130 legacy=0x7f5bb00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.261 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "leader", 2024-09-17T15:46:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 8, 2024-09-17T15:46:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 25896, 2024-09-17T15:46:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:46:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:09.261 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:46:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:46:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 25, 2024-09-17T15:46:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.262 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.264 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.265 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:09.266 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.267 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.268 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.269 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.271 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:09.272 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.274 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:09.275 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.276 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:09.278 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.279 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.280 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.281 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.282 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:09.282 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:09.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.244+0000 7f5bd6239640 1 -- 172.21.15.64:0/2497165021 >> v1:172.21.15.100:6812/4210963484 conn(0x7f5bbc040530 legacy=0x7f5bbc0429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.244+0000 7f5bd6239640 10 monclient: shutdown 2024-09-17T15:46:09.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.244+0000 7f5bd6239640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.244+0000 7f5bd6239640 1 -- 172.21.15.64:0/2497165021 >> v1:172.21.15.64:6792/0 conn(0x7f5bd0135b50 legacy=0x7f5bd0137580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.244+0000 7f5bcffff640 1 -- 172.21.15.64:0/2497165021 reap_dead start 2024-09-17T15:46:09.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.244+0000 7f5bcffff640 1 -- 172.21.15.64:0/2497165021 reap_dead start 2024-09-17T15:46:09.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.244+0000 7f5bd6239640 1 -- 172.21.15.64:0/2497165021 shutdown_connections 2024-09-17T15:46:09.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.244+0000 7f5bd6239640 1 -- 172.21.15.64:0/2497165021 >> 172.21.15.64:0/2497165021 conn(0x7f5bd006db30 msgr2=0x7f5bd00a8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.244+0000 7f5bd6239640 1 -- 172.21.15.64:0/2497165021 shutdown_connections 2024-09-17T15:46:09.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.244+0000 7f5bd6239640 1 -- 172.21.15.64:0/2497165021 wait complete. 2024-09-17T15:46:09.284 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:46:09.284 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-17T15:46:09.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:46:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.389+0000 7f12049e2640 10 monclient: get_monmap_and_config 2024-09-17T15:46:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.389+0000 7f12049e2640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.389+0000 7f12049e2640 10 monclient: monmap: 2024-09-17T15:46:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:09.390545+0000 2024-09-17T15:46:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:09.390545+0000 2024-09-17T15:46:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:09.392 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.390+0000 7f12049e2640 1 Processor -- start 2024-09-17T15:46:09.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.390+0000 7f12049e2640 1 -- start start 2024-09-17T15:46:09.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.390+0000 7f12049e2640 10 monclient: init 2024-09-17T15:46:09.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.390+0000 7f12049e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.390+0000 7f12049e2640 10 monclient: _add_conns ranks=[8,4,5,0,3,6,1,7,2] 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.390+0000 7f12049e2640 10 monclient(hunting): picked mon.noname-e con 0x7f12000a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.390+0000 7f12049e2640 10 monclient(hunting): picked mon.noname-c con 0x7f1200135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.390+0000 7f12049e2640 10 monclient(hunting): picked mon.noname-h con 0x7f1200135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.390+0000 7f12049e2640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f120009d6f0 con 0x7f1200135b50 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.390+0000 7f12049e2640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f120009d9d0 con 0x7f1200135f70 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.390+0000 7f12049e2640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f120009dcb0 con 0x7f12000a5520 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.390+0000 7f12049e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f12049e2640 10 monclient(hunting): authenticate will time out at 767.066012s 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fe59c640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f12000a5520 0x7f12000a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38716/0 (socket says 172.21.15.64:38716) 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fe59c640 1 -- 172.21.15.64:0/3968556315 learned_addr learned my addr 172.21.15.64:0/3968556315 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 1 -- 172.21.15.64:0/3968556315 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4068310293 0 0) 0x7f120009d6f0 con 0x7f1200135b50 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient(hunting): my global_id is 4504 2024-09-17T15:46:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 1 -- 172.21.15.64:0/3968556315 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11e8003520 con 0x7f1200135b50 2024-09-17T15:46:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 1 -- 172.21.15.64:0/3968556315 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3272962491 0 0) 0x7f120009dcb0 con 0x7f12000a5520 2024-09-17T15:46:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient(hunting): my global_id is 4409 2024-09-17T15:46:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 1 -- 172.21.15.64:0/3968556315 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f120009d6f0 con 0x7f12000a5520 2024-09-17T15:46:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 1 -- 172.21.15.64:0/3968556315 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 719092554 0 0) 0x7f11e8003520 con 0x7f1200135b50 2024-09-17T15:46:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 1 -- 172.21.15.64:0/3968556315 >> v1:172.21.15.64:6793/0 conn(0x7f12000a5520 legacy=0x7f12000a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 1 -- 172.21.15.64:0/3968556315 >> v1:172.21.15.100:6791/0 conn(0x7f1200135f70 legacy=0x7f12001415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 1 monclient: found mon.noname-c 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 1 -- 172.21.15.64:0/3968556315 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120009ccd0 con 0x7f1200135b50 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 1 -- 172.21.15.64:0/3968556315 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f11f4001830 con 0x7f1200135b50 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient: dump: 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.402 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.403 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.403 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.403 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.403 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.403 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.403 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.403 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.403 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.403 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.403 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.403 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f12049e2640 5 monclient: authenticate success, global_id 4504 2024-09-17T15:46:09.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f12049e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:09.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 1 -- 172.21.15.64:0/3968556315 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f11f40019d0 con 0x7f1200135b50 2024-09-17T15:46:09.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 1 -- 172.21.15.64:0/3968556315 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f11f4003950 con 0x7f1200135b50 2024-09-17T15:46:09.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:09.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f11fd59a640 10 monclient: dump: 2024-09-17T15:46:09.404 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.404 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.404 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.404 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.405 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.405 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.405 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.405 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.405 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.405 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.405 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.405 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.405 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.405 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.405 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.405 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f12049e2640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:09.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.391+0000 7f12049e2640 4 set_mon_vals no callback set 2024-09-17T15:46:09.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.392+0000 7f12049e2640 10 monclient: shutdown 2024-09-17T15:46:09.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.392+0000 7f12049e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.392+0000 7f12049e2640 1 -- 172.21.15.64:0/3968556315 >> v1:172.21.15.64:6791/0 conn(0x7f1200135b50 legacy=0x7f120013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.392+0000 7f12049e2640 1 -- 172.21.15.64:0/3968556315 shutdown_connections 2024-09-17T15:46:09.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.392+0000 7f12049e2640 1 -- 172.21.15.64:0/3968556315 >> 172.21.15.64:0/3968556315 conn(0x7f120006db30 msgr2=0x7f12000ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.392+0000 7f12049e2640 1 -- 172.21.15.64:0/3968556315 shutdown_connections 2024-09-17T15:46:09.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.392+0000 7f12049e2640 1 -- 172.21.15.64:0/3968556315 wait complete. 2024-09-17T15:46:09.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 10 monclient: monmap: 2024-09-17T15:46:09.406 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:09.407 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:09.407 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:09.407 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:09.407 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.407 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:09.407 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:09.407 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:09.407 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:09.407 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:09.407 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:09.407 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:09.407 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:09.407 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:09.408 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:09.408 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 1 Processor -- start 2024-09-17T15:46:09.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 1 -- start start 2024-09-17T15:46:09.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 10 monclient: init 2024-09-17T15:46:09.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 10 monclient: _add_conns ranks=[0,4,8,1,6,2,5,7,3] 2024-09-17T15:46:09.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 10 monclient(hunting): picked mon.noname-a con 0x7f12000a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:09.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 10 monclient(hunting): picked mon.noname-c con 0x7f1200135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:09.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 10 monclient(hunting): picked mon.noname-e con 0x7f1200135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:09.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12000a0620 con 0x7f12000a5520 2024-09-17T15:46:09.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1200134f30 con 0x7f1200135b50 2024-09-17T15:46:09.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12000a9280 con 0x7f1200135f70 2024-09-17T15:46:09.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f12049e2640 10 monclient(hunting): authenticate will time out at 767.068859s 2024-09-17T15:46:09.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f11fdd9b640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f1200135b50 0x7f12001d2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:36892/0 (socket says 172.21.15.64:36892) 2024-09-17T15:46:09.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f11fe59c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f12000a5520 0x7f12000a4b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59054/0 (socket says 172.21.15.64:59054) 2024-09-17T15:46:09.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.393+0000 7f11fdd9b640 1 -- 172.21.15.64:0/4198381258 learned_addr learned my addr 172.21.15.64:0/4198381258 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:09.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3175341826 0 0) 0x7f1200134f30 con 0x7f1200135b50 2024-09-17T15:46:09.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): my global_id is 4513 2024-09-17T15:46:09.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11e0003520 con 0x7f1200135b50 2024-09-17T15:46:09.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1948359516 0 0) 0x7f12000a9280 con 0x7f1200135f70 2024-09-17T15:46:09.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): my global_id is 4418 2024-09-17T15:46:09.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1200134f30 con 0x7f1200135f70 2024-09-17T15:46:09.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3998943657 0 0) 0x7f12000a0620 con 0x7f12000a5520 2024-09-17T15:46:09.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): my global_id is 4374 2024-09-17T15:46:09.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12000a9280 con 0x7f12000a5520 2024-09-17T15:46:09.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1401311110 0 0) 0x7f12000a9280 con 0x7f12000a5520 2024-09-17T15:46:09.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12000a0620 con 0x7f12000a5520 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2948471871 0 0) 0x7f11e0003520 con 0x7f1200135b50 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12000a9280 con 0x7f1200135b50 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3140805351 0 0) 0x7f1200134f30 con 0x7f1200135f70 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11e0003520 con 0x7f1200135f70 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f11f0001550 con 0x7f12000a5520 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): dump: 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.411 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f11f40017f0 con 0x7f1200135b50 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): dump: 2024-09-17T15:46:09.412 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.413 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.413 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.413 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.413 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.413 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.413 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.413 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.413 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.413 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.413 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.414 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.414 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.414 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.414 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.414 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f11f8001cb0 con 0x7f1200135f70 2024-09-17T15:46:09.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:09.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:09.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient(hunting): dump: 2024-09-17T15:46:09.414 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.414 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.414 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.414 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1254868255 0 0) 0x7f12000a0620 con 0x7f12000a5520 2024-09-17T15:46:09.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 >> v1:172.21.15.64:6793/0 conn(0x7f1200135f70 legacy=0x7f12001d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 >> v1:172.21.15.64:6791/0 conn(0x7f1200135b50 legacy=0x7f12001d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 monclient: found mon.a 2024-09-17T15:46:09.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:09.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12000a8d30 con 0x7f12000a5520 2024-09-17T15:46:09.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f12049e2640 5 monclient: authenticate success, global_id 4374 2024-09-17T15:46:09.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f12049e2640 10 monclient: _renew_subs 2024-09-17T15:46:09.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f12049e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:09.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f12049e2640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f12000a0340 con 0x7f12000a5520 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f12049e2640 10 monclient: _renew_subs 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f12049e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f12049e2640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f12000a5270 con 0x7f12000a5520 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f11f0001770 con 0x7f12000a5520 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f11f0004dd0 con 0x7f12000a5520 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11fcd99640 4 set_mon_vals no callback set 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.394+0000 7f11eeffd640 10 monclient: dump: 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.417 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.395+0000 7f12049e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.395+0000 7f12049e2640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f11e40065c0 con 0x7f11e4002860 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.395+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f11f00116b0 con 0x7f12000a5520 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (unknown 1990272338 0 0) 0x7f11f004eaa0 con 0x7f12000a5520 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3980556937 0 0) 0x7f11e40065c0 con 0x7f11e4002860 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 10 monclient(hunting): my global_id is 4412 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12000a0620 con 0x7f11e4002860 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11fed9d640 10 monclient: get_auth_request con 0x7f11e0040be0 auth_method 0 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 993585529 0 0) 0x7f12000a0620 con 0x7f11e4002860 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11e40065c0 con 0x7f11e4002860 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f11f4003f90 con 0x7f11e4002860 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 10 monclient: dump: 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.420 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.421 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.421 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.421 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2559548772 0 0) 0x7f11e40065c0 con 0x7f11e4002860 2024-09-17T15:46:09.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f11e400a160 con 0x7f11e4002860 2024-09-17T15:46:09.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f11f4003a00 con 0x7f11e4002860 2024-09-17T15:46:09.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:09.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 >> v1:172.21.15.64:6790/0 conn(0x7f11e4002860 legacy=0x7f11e4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.396+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 >> v1:172.21.15.64:6790/0 conn(0x7f11e4002860 legacy=0x7f11e4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f12049e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:09.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f12049e2640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f11e4009e20 con 0x7f11e40090e0 2024-09-17T15:46:09.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1647703020 0 0) 0x7f11e4009e20 con 0x7f11e40090e0 2024-09-17T15:46:09.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 10 monclient(hunting): my global_id is 4421 2024-09-17T15:46:09.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11e40065c0 con 0x7f11e40090e0 2024-09-17T15:46:09.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4264682775 0 0) 0x7f11e40065c0 con 0x7f11e40090e0 2024-09-17T15:46:09.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11e4009e20 con 0x7f11e40090e0 2024-09-17T15:46:09.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f11f4004860 con 0x7f11e40090e0 2024-09-17T15:46:09.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:09.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 10 monclient: dump: 2024-09-17T15:46:09.424 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.424 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.424 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.424 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.424 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.424 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.424 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.424 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.424 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.424 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.425 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.425 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.425 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.425 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.425 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.425 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4012914770 0 0) 0x7f11e4009e20 con 0x7f11e40090e0 2024-09-17T15:46:09.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 --> v1:172.21.15.64:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f11e400c3f0 con 0x7f11e40090e0 2024-09-17T15:46:09.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 1141407127) 0x7f11f4004940 con 0x7f11e40090e0 2024-09-17T15:46:09.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:09.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:09.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 >> v1:172.21.15.64:6790/0 conn(0x7f11e40090e0 legacy=0x7f11e4009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.403+0000 7f11eeffd640 1 -- 172.21.15.64:0/4198381258 >> v1:172.21.15.64:6790/0 conn(0x7f11e40090e0 legacy=0x7f11e4009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.426 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:09.426 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:09.426 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:09.426 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:46:09.426 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 8, 2024-09-17T15:46:09.426 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 26047, 2024-09-17T15:46:09.426 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:09.426 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:09.426 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:09.426 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:09.426 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 25, 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.427 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.428 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.428 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.428 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.428 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.428 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.428 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.428 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.428 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.428 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:09.428 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:09.428 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.428 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.428 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:09.429 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.430 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.431 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.431 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.431 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.431 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.431 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.431 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.431 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.431 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.431 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.431 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.431 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:09.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.431 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:09.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:09.432 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:09.432 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.432 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.432 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.432 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:09.432 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.433 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:09.435 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.436 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.439 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.441 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.442 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.443 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.443 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:09.443 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.443 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.443 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:09.443 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:09.443 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:09.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.445 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.445 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:09.445 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:09.445 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.445 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.445 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:09.445 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:09.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.404+0000 7f12049e2640 1 -- 172.21.15.64:0/4198381258 >> v1:172.21.15.100:6812/4210963484 conn(0x7f11e0040be0 legacy=0x7f11e00430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.404+0000 7f12049e2640 10 monclient: shutdown 2024-09-17T15:46:09.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.404+0000 7f12049e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.404+0000 7f12049e2640 1 -- 172.21.15.64:0/4198381258 >> v1:172.21.15.64:6789/0 conn(0x7f12000a5520 legacy=0x7f12000a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.404+0000 7f11fed9d640 1 -- 172.21.15.64:0/4198381258 reap_dead start 2024-09-17T15:46:09.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.404+0000 7f11fed9d640 1 -- 172.21.15.64:0/4198381258 reap_dead start 2024-09-17T15:46:09.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.404+0000 7f12049e2640 1 -- 172.21.15.64:0/4198381258 shutdown_connections 2024-09-17T15:46:09.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.404+0000 7f12049e2640 1 -- 172.21.15.64:0/4198381258 >> 172.21.15.64:0/4198381258 conn(0x7f120006db30 msgr2=0x7f120012c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.404+0000 7f12049e2640 1 -- 172.21.15.64:0/4198381258 shutdown_connections 2024-09-17T15:46:09.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.404+0000 7f12049e2640 1 -- 172.21.15.64:0/4198381258 wait complete. 2024-09-17T15:46:09.447 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:46:09.448 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-17T15:46:09.448 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:46:09.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.548+0000 7f4a4cafe640 10 monclient: get_monmap_and_config 2024-09-17T15:46:09.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.548+0000 7f4a4cafe640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.548+0000 7f4a4cafe640 10 monclient: monmap: 2024-09-17T15:46:09.550 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:09.550 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.550 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:09.549127+0000 2024-09-17T15:46:09.550 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:09.549127+0000 2024-09-17T15:46:09.550 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.550 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.550 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:09.550 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:09.550 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:09.550 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:09.551 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:09.551 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:09.551 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:09.551 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:09.551 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:09.551 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4cafe640 1 Processor -- start 2024-09-17T15:46:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4cafe640 1 -- start start 2024-09-17T15:46:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4cafe640 10 monclient: init 2024-09-17T15:46:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4cafe640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4cafe640 10 monclient: _add_conns ranks=[6,1,3,5,0,2,4,7,8] 2024-09-17T15:46:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4cafe640 10 monclient(hunting): picked mon.noname-d con 0x7f4a480a54b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4cafe640 10 monclient(hunting): picked mon.noname-f con 0x7f4a48135c80 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4cafe640 10 monclient(hunting): picked mon.noname-g con 0x7f4a481360a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4cafe640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a480aacb0 con 0x7f4a48135c80 2024-09-17T15:46:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4cafe640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a4809d780 con 0x7f4a481360a0 2024-09-17T15:46:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4cafe640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a4809da60 con 0x7f4a480a54b0 2024-09-17T15:46:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4cafe640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4cafe640 10 monclient(hunting): authenticate will time out at 767.224722s 2024-09-17T15:46:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4659c640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f4a480a54b0 0x7f4a480a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:49556/0 (socket says 172.21.15.64:49556) 2024-09-17T15:46:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.549+0000 7f4a4659c640 1 -- 172.21.15.64:0/1157924900 learned_addr learned my addr 172.21.15.64:0/1157924900 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 1 -- 172.21.15.64:0/1157924900 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 979744202 0 0) 0x7f4a4809da60 con 0x7f4a480a54b0 2024-09-17T15:46:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient(hunting): my global_id is 4380 2024-09-17T15:46:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 1 -- 172.21.15.64:0/1157924900 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a34003520 con 0x7f4a480a54b0 2024-09-17T15:46:09.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 1 -- 172.21.15.64:0/1157924900 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 424776245 0 0) 0x7f4a34003520 con 0x7f4a480a54b0 2024-09-17T15:46:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 1 -- 172.21.15.64:0/1157924900 >> v1:172.21.15.100:6790/0 conn(0x7f4a481360a0 legacy=0x7f4a48141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 1 -- 172.21.15.64:0/1157924900 >> v1:172.21.15.100:6789/0 conn(0x7f4a48135c80 legacy=0x7f4a4813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 1 monclient: found mon.noname-d 2024-09-17T15:46:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:46:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 1 -- 172.21.15.64:0/1157924900 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4809dd40 con 0x7f4a480a54b0 2024-09-17T15:46:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a4cafe640 5 monclient: authenticate success, global_id 4380 2024-09-17T15:46:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 1 -- 172.21.15.64:0/1157924900 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a38001830 con 0x7f4a480a54b0 2024-09-17T15:46:09.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a4cafe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient: dump: 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a4cafe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 1 -- 172.21.15.64:0/1157924900 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4a38001a50 con 0x7f4a480a54b0 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 1 -- 172.21.15.64:0/1157924900 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a38004be0 con 0x7f4a480a54b0 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a4cafe640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a4cafe640 4 set_mon_vals no callback set 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:09.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a45d9b640 10 monclient: dump: 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.563 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.564 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a4cafe640 10 monclient: shutdown 2024-09-17T15:46:09.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a4cafe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a4cafe640 1 -- 172.21.15.64:0/1157924900 >> v1:172.21.15.64:6792/0 conn(0x7f4a480a54b0 legacy=0x7f4a480a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a4cafe640 1 -- 172.21.15.64:0/1157924900 shutdown_connections 2024-09-17T15:46:09.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.550+0000 7f4a4cafe640 1 -- 172.21.15.64:0/1157924900 >> 172.21.15.64:0/1157924900 conn(0x7f4a4806daa0 msgr2=0x7f4a480ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.551+0000 7f4a4cafe640 1 -- 172.21.15.64:0/1157924900 shutdown_connections 2024-09-17T15:46:09.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.551+0000 7f4a4cafe640 1 -- 172.21.15.64:0/1157924900 wait complete. 2024-09-17T15:46:09.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.551+0000 7f4a4cafe640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.551+0000 7f4a4cafe640 10 monclient: monmap: 2024-09-17T15:46:09.564 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:09.564 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:09.564 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:09.565 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:09.565 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.565 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:09.565 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:09.565 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:09.565 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:09.565 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:09.565 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:09.565 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:09.565 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:09.566 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:09.566 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:09.566 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4cafe640 1 Processor -- start 2024-09-17T15:46:09.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4cafe640 1 -- start start 2024-09-17T15:46:09.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4cafe640 10 monclient: init 2024-09-17T15:46:09.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4cafe640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4cafe640 10 monclient: _add_conns ranks=[8,6,1,4,3,5,2,7,0] 2024-09-17T15:46:09.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4cafe640 10 monclient(hunting): picked mon.noname-e con 0x7f4a480a54b0 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:09.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4cafe640 10 monclient(hunting): picked mon.noname-d con 0x7f4a48135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:09.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4cafe640 10 monclient(hunting): picked mon.noname-f con 0x7f4a481360a0 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:09.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4cafe640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a480a13f0 con 0x7f4a481360a0 2024-09-17T15:46:09.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4cafe640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a48135060 con 0x7f4a48135c80 2024-09-17T15:46:09.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4cafe640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a480a2eb0 con 0x7f4a480a54b0 2024-09-17T15:46:09.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4cafe640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4cafe640 10 monclient(hunting): authenticate will time out at 767.227671s 2024-09-17T15:46:09.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4659c640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f4a480a54b0 0x7f4a4809f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38734/0 (socket says 172.21.15.64:38734) 2024-09-17T15:46:09.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a4659c640 1 -- 172.21.15.64:0/118650766 learned_addr learned my addr 172.21.15.64:0/118650766 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:09.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3014864632 0 0) 0x7f4a480a2eb0 con 0x7f4a480a54b0 2024-09-17T15:46:09.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a3effd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a3effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a3effd640 10 monclient(hunting): my global_id is 4427 2024-09-17T15:46:09.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a28003520 con 0x7f4a480a54b0 2024-09-17T15:46:09.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114318332 0 0) 0x7f4a48135060 con 0x7f4a48135c80 2024-09-17T15:46:09.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a3effd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a3effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a3effd640 10 monclient(hunting): my global_id is 4389 2024-09-17T15:46:09.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.552+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a480a2eb0 con 0x7f4a48135c80 2024-09-17T15:46:09.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 911808865 0 0) 0x7f4a480a13f0 con 0x7f4a481360a0 2024-09-17T15:46:09.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): my global_id is 4483 2024-09-17T15:46:09.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a48135060 con 0x7f4a481360a0 2024-09-17T15:46:09.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4073855089 0 0) 0x7f4a28003520 con 0x7f4a480a54b0 2024-09-17T15:46:09.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a480a13f0 con 0x7f4a480a54b0 2024-09-17T15:46:09.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a38004b20 con 0x7f4a480a54b0 2024-09-17T15:46:09.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:09.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:46:09.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): dump: 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.570 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.571 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.571 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:09.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1764839335 0 0) 0x7f4a480a2eb0 con 0x7f4a48135c80 2024-09-17T15:46:09.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a28003520 con 0x7f4a48135c80 2024-09-17T15:46:09.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a30001990 con 0x7f4a48135c80 2024-09-17T15:46:09.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:09.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:09.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): dump: 2024-09-17T15:46:09.571 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.571 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.571 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.571 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3365165077 0 0) 0x7f4a48135060 con 0x7f4a481360a0 2024-09-17T15:46:09.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a480a2eb0 con 0x7f4a481360a0 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a40001b40 con 0x7f4a481360a0 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient(hunting): dump: 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.573 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3783843939 0 0) 0x7f4a480a13f0 con 0x7f4a480a54b0 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 >> v1:172.21.15.64:6792/0 conn(0x7f4a48135c80 legacy=0x7f4a480a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 >> v1:172.21.15.100:6789/0 conn(0x7f4a481360a0 legacy=0x7f4a480a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 monclient: found mon.e 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:09.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a480a3b70 con 0x7f4a480a54b0 2024-09-17T15:46:09.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a4cafe640 5 monclient: authenticate success, global_id 4427 2024-09-17T15:46:09.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a4cafe640 10 monclient: _renew_subs 2024-09-17T15:46:09.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a4cafe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:09.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a4cafe640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4a480a5120 con 0x7f4a480a54b0 2024-09-17T15:46:09.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a4cafe640 10 monclient: _renew_subs 2024-09-17T15:46:09.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a4cafe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:09.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a4cafe640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f4a481d7010 con 0x7f4a480a54b0 2024-09-17T15:46:09.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4a380011b0 con 0x7f4a480a54b0 2024-09-17T15:46:09.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a38001950 con 0x7f4a480a54b0 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a4559a640 4 set_mon_vals no callback set 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.553+0000 7f4a3effd640 10 monclient: dump: 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.576 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.554+0000 7f4a4cafe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.554+0000 7f4a4cafe640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4a2c0065c0 con 0x7f4a2c002860 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.554+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f4a38011960 con 0x7f4a480a54b0 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.554+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (unknown 1990272338 0 0) 0x7f4a3804ed50 con 0x7f4a480a54b0 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.554+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2922827725 0 0) 0x7f4a2c0065c0 con 0x7f4a2c002860 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.554+0000 7f4a3effd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.554+0000 7f4a3effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.554+0000 7f4a3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.554+0000 7f4a3effd640 10 monclient(hunting): my global_id is 4522 2024-09-17T15:46:09.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.554+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a480a13f0 con 0x7f4a2c002860 2024-09-17T15:46:09.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a46d9d640 10 monclient: get_auth_request con 0x7f4a28040c30 auth_method 0 2024-09-17T15:46:09.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3982245105 0 0) 0x7f4a480a13f0 con 0x7f4a2c002860 2024-09-17T15:46:09.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a2c0065c0 con 0x7f4a2c002860 2024-09-17T15:46:09.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a30003c00 con 0x7f4a2c002860 2024-09-17T15:46:09.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:09.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a3effd640 10 monclient: dump: 2024-09-17T15:46:09.578 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.578 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.578 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.578 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3095104115 0 0) 0x7f4a2c0065c0 con 0x7f4a2c002860 2024-09-17T15:46:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a2c00a160 con 0x7f4a2c002860 2024-09-17T15:46:09.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4a30004000 con 0x7f4a2c002860 2024-09-17T15:46:09.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a3effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a3effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:09.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 >> v1:172.21.15.64:6791/0 conn(0x7f4a2c002860 legacy=0x7f4a2c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.555+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 >> v1:172.21.15.64:6791/0 conn(0x7f4a2c002860 legacy=0x7f4a2c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.561+0000 7f4a4cafe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.561+0000 7f4a4cafe640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4a2c009e50 con 0x7f4a2c009110 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3524788930 0 0) 0x7f4a2c009e50 con 0x7f4a2c009110 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 10 monclient(hunting): my global_id is 4531 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a2c0065c0 con 0x7f4a2c009110 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3303934543 0 0) 0x7f4a2c0065c0 con 0x7f4a2c009110 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a2c009e50 con 0x7f4a2c009110 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a30004770 con 0x7f4a2c009110 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 10 monclient: dump: 2024-09-17T15:46:09.582 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.583 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.584 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1861853265 0 0) 0x7f4a2c009e50 con 0x7f4a2c009110 2024-09-17T15:46:09.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 --> v1:172.21.15.64:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4a2c00c3f0 con 0x7f4a2c009110 2024-09-17T15:46:09.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 647518066) 0x7f4a300069b0 con 0x7f4a2c009110 2024-09-17T15:46:09.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:09.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:09.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 >> v1:172.21.15.64:6791/0 conn(0x7f4a2c009110 legacy=0x7f4a2c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.562+0000 7f4a3effd640 1 -- 172.21.15.64:0/118650766 >> v1:172.21.15.64:6791/0 conn(0x7f4a2c009110 legacy=0x7f4a2c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.584 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:09.584 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:46:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 8, 2024-09-17T15:46:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 26219, 2024-09-17T15:46:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:46:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:46:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:46:09.585 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 25, 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.586 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.587 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:09.588 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.590 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.591 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.593 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:09.594 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:09.596 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.598 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:09.600 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.601 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.602 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:46:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.603 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:09.603 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:09.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.563+0000 7f4a4cafe640 1 -- 172.21.15.64:0/118650766 >> v1:172.21.15.100:6812/4210963484 conn(0x7f4a28040c30 legacy=0x7f4a280430f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.563+0000 7f4a4cafe640 10 monclient: shutdown 2024-09-17T15:46:09.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.563+0000 7f4a4cafe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.563+0000 7f4a4cafe640 1 -- 172.21.15.64:0/118650766 >> v1:172.21.15.64:6793/0 conn(0x7f4a480a54b0 legacy=0x7f4a4809f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.563+0000 7f4a46d9d640 1 -- 172.21.15.64:0/118650766 reap_dead start 2024-09-17T15:46:09.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.563+0000 7f4a46d9d640 1 -- 172.21.15.64:0/118650766 reap_dead start 2024-09-17T15:46:09.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.563+0000 7f4a4cafe640 1 -- 172.21.15.64:0/118650766 shutdown_connections 2024-09-17T15:46:09.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.563+0000 7f4a4cafe640 1 -- 172.21.15.64:0/118650766 >> 172.21.15.64:0/118650766 conn(0x7f4a4806daa0 msgr2=0x7f4a480a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.563+0000 7f4a4cafe640 1 -- 172.21.15.64:0/118650766 shutdown_connections 2024-09-17T15:46:09.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.563+0000 7f4a4cafe640 1 -- 172.21.15.64:0/118650766 wait complete. 2024-09-17T15:46:09.606 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:46:09.606 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-17T15:46:09.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-17T15:46:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.706+0000 7f2cec132640 10 monclient: get_monmap_and_config 2024-09-17T15:46:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.706+0000 7f2cec132640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.706+0000 7f2cec132640 10 monclient: monmap: 2024-09-17T15:46:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:09.707351+0000 2024-09-17T15:46:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:09.707351+0000 2024-09-17T15:46:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.708 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:09.709 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:09.709 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.707+0000 7f2cec132640 1 Processor -- start 2024-09-17T15:46:09.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.707+0000 7f2cec132640 1 -- start start 2024-09-17T15:46:09.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.707+0000 7f2cec132640 10 monclient: init 2024-09-17T15:46:09.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.707+0000 7f2cec132640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.707+0000 7f2cec132640 10 monclient: _add_conns ranks=[1,8,3,0,5,4,6,7,2] 2024-09-17T15:46:09.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.707+0000 7f2cec132640 10 monclient(hunting): picked mon.noname-f con 0x7f2ce40a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:09.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.707+0000 7f2cec132640 10 monclient(hunting): picked mon.noname-e con 0x7f2ce4135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:09.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.707+0000 7f2cec132640 10 monclient(hunting): picked mon.noname-g con 0x7f2ce4135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:09.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.707+0000 7f2cec132640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ce409d6f0 con 0x7f2ce40a5520 2024-09-17T15:46:09.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.707+0000 7f2cec132640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ce409d9d0 con 0x7f2ce4135f70 2024-09-17T15:46:09.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.707+0000 7f2cec132640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ce409dcb0 con 0x7f2ce4135b50 2024-09-17T15:46:09.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.707+0000 7f2cec132640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.707+0000 7f2cec132640 10 monclient(hunting): authenticate will time out at 767.382972s 2024-09-17T15:46:09.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce96cd640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f2ce4135b50 0x7f2ce413deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38740/0 (socket says 172.21.15.64:38740) 2024-09-17T15:46:09.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce96cd640 1 -- 172.21.15.64:0/1563628586 learned_addr learned my addr 172.21.15.64:0/1563628586 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 1 -- 172.21.15.64:0/1563628586 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3445878811 0 0) 0x7f2ce409dcb0 con 0x7f2ce4135b50 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient(hunting): my global_id is 4436 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 1 -- 172.21.15.64:0/1563628586 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cd0003520 con 0x7f2ce4135b50 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 1 -- 172.21.15.64:0/1563628586 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4175924938 0 0) 0x7f2cd0003520 con 0x7f2ce4135b50 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 1 -- 172.21.15.64:0/1563628586 >> v1:172.21.15.100:6790/0 conn(0x7f2ce4135f70 legacy=0x7f2ce41415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 1 -- 172.21.15.64:0/1563628586 >> v1:172.21.15.100:6789/0 conn(0x7f2ce40a5520 legacy=0x7f2ce40a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 1 monclient: found mon.noname-e 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:46:09.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 1 -- 172.21.15.64:0/1563628586 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce409ccd0 con 0x7f2ce4135b50 2024-09-17T15:46:09.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 1 -- 172.21.15.64:0/1563628586 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2cd4001830 con 0x7f2ce4135b50 2024-09-17T15:46:09.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:46:09.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient: dump: 2024-09-17T15:46:09.719 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.719 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.719 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.719 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.719 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.719 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2cec132640 5 monclient: authenticate success, global_id 4436 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2cec132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 1 -- 172.21.15.64:0/1563628586 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2cd40019d0 con 0x7f2ce4135b50 2024-09-17T15:46:09.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2cec132640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2cec132640 4 set_mon_vals no callback set 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 1 -- 172.21.15.64:0/1563628586 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2cd4004be0 con 0x7f2ce4135b50 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.708+0000 7f2ce8ecc640 10 monclient: dump: 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.709+0000 7f2cec132640 10 monclient: shutdown 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.709+0000 7f2cec132640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.709+0000 7f2cec132640 1 -- 172.21.15.64:0/1563628586 >> v1:172.21.15.64:6793/0 conn(0x7f2ce4135b50 legacy=0x7f2ce413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.709+0000 7f2cec132640 1 -- 172.21.15.64:0/1563628586 shutdown_connections 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.709+0000 7f2cec132640 1 -- 172.21.15.64:0/1563628586 >> 172.21.15.64:0/1563628586 conn(0x7f2ce406db30 msgr2=0x7f2ce40ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.709+0000 7f2cec132640 1 -- 172.21.15.64:0/1563628586 shutdown_connections 2024-09-17T15:46:09.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.709+0000 7f2cec132640 1 -- 172.21.15.64:0/1563628586 wait complete. 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.709+0000 7f2cec132640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.709+0000 7f2cec132640 10 monclient: monmap: 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:09.723 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cec132640 1 Processor -- start 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cec132640 1 -- start start 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cec132640 10 monclient: init 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cec132640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cec132640 10 monclient: _add_conns ranks=[2,4,8,6,5,7,0,3,1] 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cec132640 10 monclient(hunting): picked mon.noname-b con 0x7f2ce40a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cec132640 10 monclient(hunting): picked mon.noname-c con 0x7f2ce4135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cec132640 10 monclient(hunting): picked mon.noname-e con 0x7f2ce4135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cec132640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ce413b120 con 0x7f2ce40a5520 2024-09-17T15:46:09.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cec132640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ce4134ad0 con 0x7f2ce4135b50 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cec132640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ce40a9320 con 0x7f2ce4135f70 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cec132640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cec132640 10 monclient(hunting): authenticate will time out at 767.385588s 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2ce9ece640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f2ce40a5520 0x7f2ce4136f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46906/0 (socket says 172.21.15.64:46906) 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2ce96cd640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f2ce4135b50 0x7f2ce4137610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:36914/0 (socket says 172.21.15.64:36914) 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2ce9ece640 1 -- 172.21.15.64:0/4164872128 learned_addr learned my addr 172.21.15.64:0/4164872128 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cea6cf640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f2ce4135f70 0x7f2ce41d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38750/0 (socket says 172.21.15.64:38750) 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1268309551 0 0) 0x7f2ce40a9320 con 0x7f2ce4135f70 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cda7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cda7fc640 10 monclient(hunting): my global_id is 4445 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cc8003550 con 0x7f2ce4135f70 2024-09-17T15:46:09.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3261328558 0 0) 0x7f2ce409d6f0 con 0x7f2ce40a5520 2024-09-17T15:46:09.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cda7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cda7fc640 10 monclient(hunting): my global_id is 4430 2024-09-17T15:46:09.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.710+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ce40a9320 con 0x7f2ce40a5520 2024-09-17T15:46:09.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3816698338 0 0) 0x7f2ce4134ad0 con 0x7f2ce4135b50 2024-09-17T15:46:09.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): my global_id is 4540 2024-09-17T15:46:09.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ce409d6f0 con 0x7f2ce4135b50 2024-09-17T15:46:09.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4047074652 0 0) 0x7f2cc8003550 con 0x7f2ce4135f70 2024-09-17T15:46:09.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ce4134ad0 con 0x7f2ce4135f70 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2cdc001d70 con 0x7f2ce4135f70 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): dump: 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.727 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.728 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.728 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.728 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.728 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.728 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:09.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2059538249 0 0) 0x7f2ce40a9320 con 0x7f2ce40a5520 2024-09-17T15:46:09.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2cc8003550 con 0x7f2ce40a5520 2024-09-17T15:46:09.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3860568364 0 0) 0x7f2ce409d6f0 con 0x7f2ce4135b50 2024-09-17T15:46:09.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ce40a9320 con 0x7f2ce4135b50 2024-09-17T15:46:09.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2ce0001ad0 con 0x7f2ce40a5520 2024-09-17T15:46:09.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:09.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): dump: 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.729 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2cd4004b20 con 0x7f2ce4135b50 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient(hunting): dump: 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.730 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3131274307 0 0) 0x7f2ce4134ad0 con 0x7f2ce4135f70 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 >> v1:172.21.15.64:6791/0 conn(0x7f2ce4135b50 legacy=0x7f2ce4137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 >> v1:172.21.15.64:6790/0 conn(0x7f2ce40a5520 legacy=0x7f2ce4136f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 monclient: found mon.e 2024-09-17T15:46:09.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:09.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce41389a0 con 0x7f2ce4135f70 2024-09-17T15:46:09.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cec132640 5 monclient: authenticate success, global_id 4445 2024-09-17T15:46:09.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cec132640 10 monclient: _renew_subs 2024-09-17T15:46:09.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cec132640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:09.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cec132640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2ce4137d20 con 0x7f2ce4135f70 2024-09-17T15:46:09.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cec132640 10 monclient: _renew_subs 2024-09-17T15:46:09.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cec132640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:09.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cec132640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f2ce413a460 con 0x7f2ce4135f70 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2cdc002070 con 0x7f2ce4135f70 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cdbfff640 4 set_mon_vals no callback set 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2cdc003f80 con 0x7f2ce4135f70 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.711+0000 7f2cda7fc640 10 monclient: dump: 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.733 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.712+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f2cdc0041a0 con 0x7f2ce4135f70 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.712+0000 7f2cec132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.712+0000 7f2cec132640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ccc0065c0 con 0x7f2ccc002860 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.712+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (unknown 1990272338 0 0) 0x7f2cdc04ef90 con 0x7f2ce4135f70 2024-09-17T15:46:09.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3733016615 0 0) 0x7f2ccc0065c0 con 0x7f2ccc002860 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 10 monclient(hunting): my global_id is 4398 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ce4134ad0 con 0x7f2ccc002860 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2ce96cd640 10 monclient: get_auth_request con 0x7f2cc8040e00 auth_method 0 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3404395615 0 0) 0x7f2ce4134ad0 con 0x7f2ccc002860 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ccc0065c0 con 0x7f2ccc002860 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2ce0001c70 con 0x7f2ccc002860 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 10 monclient: dump: 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.735 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.736 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.737 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 511885512 0 0) 0x7f2ccc0065c0 con 0x7f2ccc002860 2024-09-17T15:46:09.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ccc00a160 con 0x7f2ccc002860 2024-09-17T15:46:09.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2ce00057c0 con 0x7f2ccc002860 2024-09-17T15:46:09.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:09.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 >> v1:172.21.15.64:6792/0 conn(0x7f2ccc002860 legacy=0x7f2ccc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.713+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 >> v1:172.21.15.64:6792/0 conn(0x7f2ccc002860 legacy=0x7f2ccc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.719+0000 7f2cec132640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:09.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.719+0000 7f2cec132640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ccc009e00 con 0x7f2ccc0090e0 2024-09-17T15:46:09.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 480929191 0 0) 0x7f2ccc009e00 con 0x7f2ccc0090e0 2024-09-17T15:46:09.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 10 monclient(hunting): my global_id is 4407 2024-09-17T15:46:09.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ccc0065c0 con 0x7f2ccc0090e0 2024-09-17T15:46:09.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2710591276 0 0) 0x7f2ccc0065c0 con 0x7f2ccc0090e0 2024-09-17T15:46:09.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ccc009e00 con 0x7f2ccc0090e0 2024-09-17T15:46:09.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2ce00043e0 con 0x7f2ccc0090e0 2024-09-17T15:46:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 10 monclient: dump: 2024-09-17T15:46:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.740 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.741 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.741 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.741 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.741 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.741 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.741 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2549736460 0 0) 0x7f2ccc009e00 con 0x7f2ccc0090e0 2024-09-17T15:46:09.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 --> v1:172.21.15.64:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2ccc00c3a0 con 0x7f2ccc0090e0 2024-09-17T15:46:09.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7710 (unknown 0 0 3801997856) 0x7f2ce00079d0 con 0x7f2ccc0090e0 2024-09-17T15:46:09.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:09.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:09.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 >> v1:172.21.15.64:6792/0 conn(0x7f2ccc0090e0 legacy=0x7f2ccc009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.720+0000 7f2cda7fc640 1 -- 172.21.15.64:0/4164872128 >> v1:172.21.15.64:6792/0 conn(0x7f2ccc0090e0 legacy=0x7f2ccc009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.742 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:09.742 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:09.742 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:09.742 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:46:09.742 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 8, 2024-09-17T15:46:09.742 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 26371, 2024-09-17T15:46:09.743 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:09.743 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:09.743 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:09.743 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:09.743 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:46:09.743 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:09.743 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:46:09.743 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:46:09.743 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:09.744 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:09.744 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.744 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 26, 2024-09-17T15:46:09.744 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:09.744 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:09.744 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:09.744 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.744 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.744 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.744 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.744 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.744 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.744 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.745 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.745 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.745 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.745 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.745 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.745 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:09.745 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:09.745 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.745 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.745 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.745 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.746 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.746 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.746 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.746 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.746 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.746 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.746 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.746 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.746 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:09.746 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:09.747 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:09.747 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:09.747 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:09.747 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:09.747 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:09.747 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:09.747 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:09.747 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:09.747 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:09.747 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:09.747 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:09.747 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.748 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.749 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.749 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:09.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.749 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:09.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.749 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:09.749 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:09.749 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.749 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.749 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.749 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:09.749 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.750 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.751 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.751 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:09.751 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.751 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.751 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.751 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:09.751 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:09.752 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.752 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.752 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.752 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:09.752 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:09.752 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.752 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.753 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.753 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:09.753 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.753 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.753 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:09.753 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:09.753 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.753 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.754 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.754 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:09.754 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:09.754 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.754 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.754 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.754 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:09.755 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.755 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.755 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.755 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:09.755 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:09.755 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.755 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.755 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.755 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:09.755 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:09.756 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.756 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.756 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.756 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:09.756 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.756 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.756 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.756 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:09.756 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:09.756 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.756 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:09.758 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.760 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.761 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:09.764 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:09.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.721+0000 7f2cec132640 1 -- 172.21.15.64:0/4164872128 >> v1:172.21.15.100:6812/4210963484 conn(0x7f2cc8040e00 legacy=0x7f2cc80432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.721+0000 7f2cec132640 10 monclient: shutdown 2024-09-17T15:46:09.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.721+0000 7f2cec132640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.721+0000 7f2cec132640 1 -- 172.21.15.64:0/4164872128 >> v1:172.21.15.64:6793/0 conn(0x7f2ce4135f70 legacy=0x7f2ce41d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.721+0000 7f2cea6cf640 1 -- 172.21.15.64:0/4164872128 reap_dead start 2024-09-17T15:46:09.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.721+0000 7f2cea6cf640 1 -- 172.21.15.64:0/4164872128 reap_dead start 2024-09-17T15:46:09.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.721+0000 7f2cec132640 1 -- 172.21.15.64:0/4164872128 shutdown_connections 2024-09-17T15:46:09.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.721+0000 7f2cec132640 1 -- 172.21.15.64:0/4164872128 >> 172.21.15.64:0/4164872128 conn(0x7f2ce406db30 msgr2=0x7f2ce40a8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.721+0000 7f2cec132640 1 -- 172.21.15.64:0/4164872128 shutdown_connections 2024-09-17T15:46:09.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.721+0000 7f2cec132640 1 -- 172.21.15.64:0/4164872128 wait complete. 2024-09-17T15:46:09.766 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:46:09.766 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-17T15:46:09.766 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-17T15:46:09.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.866+0000 7fbe026e6640 10 monclient: get_monmap_and_config 2024-09-17T15:46:09.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.866+0000 7fbe026e6640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.866+0000 7fbe026e6640 10 monclient: monmap: 2024-09-17T15:46:09.878 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:09.878 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.878 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:09.867033+0000 2024-09-17T15:46:09.878 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:09.867033+0000 2024-09-17T15:46:09.878 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.878 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.878 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:09.878 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.866+0000 7fbe026e6640 1 Processor -- start 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.866+0000 7fbe026e6640 1 -- start start 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.866+0000 7fbe026e6640 10 monclient: init 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbe026e6640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbe026e6640 10 monclient: _add_conns ranks=[0,4,8,2,7,3,1,6,5] 2024-09-17T15:46:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbe026e6640 10 monclient(hunting): picked mon.noname-a con 0x7fbdfc0a54b0 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbe026e6640 10 monclient(hunting): picked mon.noname-c con 0x7fbdfc135c80 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbe026e6640 10 monclient(hunting): picked mon.noname-e con 0x7fbdfc1360a0 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbe026e6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdfc0aacb0 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbe026e6640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdfc09d780 con 0x7fbdfc135c80 2024-09-17T15:46:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbe026e6640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdfc09da60 con 0x7fbdfc1360a0 2024-09-17T15:46:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbe026e6640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbe026e6640 10 monclient(hunting): authenticate will time out at 767.542491s 2024-09-17T15:46:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbe00c83640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fbdfc1360a0 0x7fbdfc141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38766/0 (socket says 172.21.15.64:38766) 2024-09-17T15:46:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbe00c83640 1 -- 172.21.15.64:0/2085436645 learned_addr learned my addr 172.21.15.64:0/2085436645 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfbfff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fbdfc0a54b0 0x7fbdfc0a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59064/0 (socket says 172.21.15.64:59064) 2024-09-17T15:46:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfaffd640 1 -- 172.21.15.64:0/2085436645 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1138556479 0 0) 0x7fbdfc0aacb0 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfaffd640 10 monclient(hunting): my global_id is 4383 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfaffd640 1 -- 172.21.15.64:0/2085436645 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbde4003520 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfaffd640 1 -- 172.21.15.64:0/2085436645 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3391358076 0 0) 0x7fbdfc09da60 con 0x7fbdfc1360a0 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfaffd640 10 monclient(hunting): my global_id is 4454 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfaffd640 1 -- 172.21.15.64:0/2085436645 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdfc0aacb0 con 0x7fbdfc1360a0 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.867+0000 7fbdfaffd640 1 -- 172.21.15.64:0/2085436645 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2383040896 0 0) 0x7fbdfc09d780 con 0x7fbdfc135c80 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient(hunting): my global_id is 4549 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 1 -- 172.21.15.64:0/2085436645 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdfc09da60 con 0x7fbdfc135c80 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 1 -- 172.21.15.64:0/2085436645 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3297428795 0 0) 0x7fbde4003520 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 1 -- 172.21.15.64:0/2085436645 >> v1:172.21.15.64:6793/0 conn(0x7fbdfc1360a0 legacy=0x7fbdfc141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 1 -- 172.21.15.64:0/2085436645 >> v1:172.21.15.64:6791/0 conn(0x7fbdfc135c80 legacy=0x7fbdfc13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 1 monclient: found mon.noname-a 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 1 -- 172.21.15.64:0/2085436645 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdfc09dd40 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 1 -- 172.21.15.64:0/2085436645 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbdec001830 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient: dump: 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbe026e6640 5 monclient: authenticate success, global_id 4383 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbe026e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 1 -- 172.21.15.64:0/2085436645 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbdec001a50 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 1 -- 172.21.15.64:0/2085436645 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbdec004b20 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbe026e6640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbe026e6640 4 set_mon_vals no callback set 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbdfaffd640 10 monclient: dump: 2024-09-17T15:46:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.886 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.886 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbe026e6640 10 monclient: shutdown 2024-09-17T15:46:09.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbe026e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbe026e6640 1 -- 172.21.15.64:0/2085436645 >> v1:172.21.15.64:6789/0 conn(0x7fbdfc0a54b0 legacy=0x7fbdfc0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbe026e6640 1 -- 172.21.15.64:0/2085436645 shutdown_connections 2024-09-17T15:46:09.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbe026e6640 1 -- 172.21.15.64:0/2085436645 >> 172.21.15.64:0/2085436645 conn(0x7fbdfc06daa0 msgr2=0x7fbdfc0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbe026e6640 1 -- 172.21.15.64:0/2085436645 shutdown_connections 2024-09-17T15:46:09.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.868+0000 7fbe026e6640 1 -- 172.21.15.64:0/2085436645 wait complete. 2024-09-17T15:46:09.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.869+0000 7fbe026e6640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.869+0000 7fbe026e6640 10 monclient: monmap: 2024-09-17T15:46:09.886 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:09.886 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:09.887 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:09.887 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:09.887 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.887 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:09.887 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:09.887 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:09.887 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:09.887 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:09.887 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:09.887 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:09.887 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:09.888 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:09.888 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:09.888 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.869+0000 7fbe026e6640 1 Processor -- start 2024-09-17T15:46:09.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbe026e6640 1 -- start start 2024-09-17T15:46:09.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbe026e6640 10 monclient: init 2024-09-17T15:46:09.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbe026e6640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbe026e6640 10 monclient: _add_conns ranks=[0,6,2,3,8,7,5,4,1] 2024-09-17T15:46:09.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbe026e6640 10 monclient(hunting): picked mon.noname-a con 0x7fbdfc0a54b0 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:09.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbe026e6640 10 monclient(hunting): picked mon.noname-d con 0x7fbdfc135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:09.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbe026e6640 10 monclient(hunting): picked mon.noname-b con 0x7fbdfc1360a0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:09.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbe026e6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdfc09f840 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbe026e6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdfc1cebf0 con 0x7fbdfc1360a0 2024-09-17T15:46:09.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbe026e6640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdfc0a6b90 con 0x7fbdfc135c80 2024-09-17T15:46:09.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbe026e6640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbe026e6640 10 monclient(hunting): authenticate will time out at 767.545682s 2024-09-17T15:46:09.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbdfbfff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fbdfc0a54b0 0x7fbdfc1cddd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59070/0 (socket says 172.21.15.64:59070) 2024-09-17T15:46:09.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbdfb7fe640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fbdfc135c80 0x7fbdfc1ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:49606/0 (socket says 172.21.15.64:49606) 2024-09-17T15:46:09.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbdfbfff640 1 -- 172.21.15.64:0/1135851776 learned_addr learned my addr 172.21.15.64:0/1135851776 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:09.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3223616265 0 0) 0x7fbdfc09f840 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbdf8ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbdf8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbdf8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbdf8ff9640 10 monclient(hunting): my global_id is 4392 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.870+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbde8003520 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1392270685 0 0) 0x7fbdfc1cebf0 con 0x7fbdfc1360a0 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): my global_id is 4439 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdfc09f840 con 0x7fbdfc1360a0 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4142482241 0 0) 0x7fbdfc0a6b90 con 0x7fbdfc135c80 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): my global_id is 4416 2024-09-17T15:46:09.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdfc1cebf0 con 0x7fbdfc135c80 2024-09-17T15:46:09.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 744723596 0 0) 0x7fbde8003520 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdfc0a6b90 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbdec0017f0 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:09.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:09.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): dump: 2024-09-17T15:46:09.891 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.891 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.892 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.892 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.892 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.892 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.892 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.892 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.893 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.893 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.893 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.893 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.893 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.893 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.893 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.893 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:09.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3411816003 0 0) 0x7fbdfc09f840 con 0x7fbdfc1360a0 2024-09-17T15:46:09.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbde8003520 con 0x7fbdfc1360a0 2024-09-17T15:46:09.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1538364362 0 0) 0x7fbdfc1cebf0 con 0x7fbdfc135c80 2024-09-17T15:46:09.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdfc09f840 con 0x7fbdfc135c80 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbdf4001cf0 con 0x7fbdfc1360a0 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): dump: 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.894 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbdf0001910 con 0x7fbdfc135c80 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient(hunting): dump: 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.895 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 153414670 0 0) 0x7fbdfc0a6b90 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 >> v1:172.21.15.64:6792/0 conn(0x7fbdfc135c80 legacy=0x7fbdfc1ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 >> v1:172.21.15.64:6790/0 conn(0x7fbdfc1360a0 legacy=0x7fbdfc1d6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 monclient: found mon.a 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdfc09fe10 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbe026e6640 5 monclient: authenticate success, global_id 4392 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbe026e6640 10 monclient: _renew_subs 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbe026e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbe026e6640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbdfc0a00f0 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbe026e6640 10 monclient: _renew_subs 2024-09-17T15:46:09.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbe026e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:09.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbe026e6640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbdfc09e390 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbdec004de0 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbdec0052d0 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:09.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdf8ff9640 10 monclient: dump: 2024-09-17T15:46:09.898 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.898 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.898 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.898 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.898 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.898 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.871+0000 7fbdfa7fc640 4 set_mon_vals no callback set 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.872+0000 7fbe026e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.872+0000 7fbe026e6640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbdfc0008d0 con 0x7fbdfc09eeb0 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.872+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fbdec011bb0 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.872+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (unknown 1990272338 0 0) 0x7fbdec04ece0 con 0x7fbdfc0a54b0 2024-09-17T15:46:09.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.872+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4158104229 0 0) 0x7fbdfc0008d0 con 0x7fbdfc09eeb0 2024-09-17T15:46:09.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.872+0000 7fbdf8ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.872+0000 7fbdf8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.872+0000 7fbdf8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.872+0000 7fbdf8ff9640 10 monclient(hunting): my global_id is 4463 2024-09-17T15:46:09.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.872+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdfc0a6b90 con 0x7fbdfc09eeb0 2024-09-17T15:46:09.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbe00c83640 10 monclient: get_auth_request con 0x7fbde8040df0 auth_method 0 2024-09-17T15:46:09.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 221040912 0 0) 0x7fbdfc0a6b90 con 0x7fbdfc09eeb0 2024-09-17T15:46:09.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdfc0008d0 con 0x7fbdfc09eeb0 2024-09-17T15:46:09.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbdf0003930 con 0x7fbdfc09eeb0 2024-09-17T15:46:09.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbdf8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbdf8ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:09.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbdf8ff9640 10 monclient: dump: 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.901 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.902 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2950760299 0 0) 0x7fbdfc0008d0 con 0x7fbdfc09eeb0 2024-09-17T15:46:09.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbdfc09f290 con 0x7fbdfc09eeb0 2024-09-17T15:46:09.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbdf00057c0 con 0x7fbdfc09eeb0 2024-09-17T15:46:09.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbdf8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbdf8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:09.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 >> v1:172.21.15.64:6793/0 conn(0x7fbdfc09eeb0 legacy=0x7fbdfc12d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.873+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 >> v1:172.21.15.64:6793/0 conn(0x7fbdfc09eeb0 legacy=0x7fbdfc12d810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.879+0000 7fbe026e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:09.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.879+0000 7fbe026e6640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbdfc142d60 con 0x7fbdfc02e450 2024-09-17T15:46:09.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 556793651 0 0) 0x7fbdfc142d60 con 0x7fbdfc02e450 2024-09-17T15:46:09.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 10 monclient(hunting): my global_id is 4472 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdfc0008d0 con 0x7fbdfc02e450 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3460386632 0 0) 0x7fbdfc0008d0 con 0x7fbdfc02e450 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdfc142d60 con 0x7fbdfc02e450 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbdf00041c0 con 0x7fbdfc02e450 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 10 monclient: dump: 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:09.905 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2798594990 0 0) 0x7fbdfc142d60 con 0x7fbdfc02e450 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 --> v1:172.21.15.64:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbdfc02e830 con 0x7fbdfc02e450 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 2665434225) 0x7fbdf0004230 con 0x7fbdfc02e450 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:09.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 >> v1:172.21.15.64:6793/0 conn(0x7fbdfc02e450 legacy=0x7fbdfc12df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.880+0000 7fbdf8ff9640 1 -- 172.21.15.64:0/1135851776 >> v1:172.21.15.64:6793/0 conn(0x7fbdfc02e450 legacy=0x7fbdfc12df20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.907 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:09.907 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:09.907 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:09.907 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:46:09.907 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 8, 2024-09-17T15:46:09.907 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 26527, 2024-09-17T15:46:09.907 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:09.907 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:09.908 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:09.908 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:09.908 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:46:09.908 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:09.908 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:46:09.908 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:46:09.908 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:09.908 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:09.908 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.908 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 26, 2024-09-17T15:46:09.908 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:09.908 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:09.908 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:09.909 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:09.910 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.910 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.910 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.910 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.910 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.910 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.910 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.910 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.910 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.910 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.910 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:09.911 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:09.912 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:09.912 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:09.912 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:09.912 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:09.912 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:09.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:09.912 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:09.912 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:09.912 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:09.912 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:09.912 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:09.912 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:09.912 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:09.913 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:09.913 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.913 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:09.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:09.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.913 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:09.913 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:09.913 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.913 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.913 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.913 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:09.914 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.914 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.914 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:09.914 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:09.914 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.914 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.914 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.914 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:09.914 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.915 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.916 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:09.917 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.918 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.920 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:09.920 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:09.920 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.920 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.920 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.920 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:09.920 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.920 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.920 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.920 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:09.920 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.921 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.922 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.922 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:09.922 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:09.922 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.922 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.922 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.922 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:09.922 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:09.922 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.922 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:09.924 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:09.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:09.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:09.926 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.926 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:09.926 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:09.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:09.926 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:09.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:09.926 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:09.926 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:09.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.881+0000 7fbe026e6640 1 -- 172.21.15.64:0/1135851776 >> v1:172.21.15.100:6812/4210963484 conn(0x7fbde8040df0 legacy=0x7fbde80432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.881+0000 7fbe026e6640 10 monclient: shutdown 2024-09-17T15:46:09.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.881+0000 7fbe026e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.881+0000 7fbe026e6640 1 -- 172.21.15.64:0/1135851776 >> v1:172.21.15.64:6789/0 conn(0x7fbdfc0a54b0 legacy=0x7fbdfc1cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.881+0000 7fbe00c83640 1 -- 172.21.15.64:0/1135851776 reap_dead start 2024-09-17T15:46:09.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.881+0000 7fbe00c83640 1 -- 172.21.15.64:0/1135851776 reap_dead start 2024-09-17T15:46:09.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.881+0000 7fbe026e6640 1 -- 172.21.15.64:0/1135851776 shutdown_connections 2024-09-17T15:46:09.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.881+0000 7fbe026e6640 1 -- 172.21.15.64:0/1135851776 >> 172.21.15.64:0/1135851776 conn(0x7fbdfc06daa0 msgr2=0x7fbdfc0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.881+0000 7fbe026e6640 1 -- 172.21.15.64:0/1135851776 shutdown_connections 2024-09-17T15:46:09.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:09.881+0000 7fbe026e6640 1 -- 172.21.15.64:0/1135851776 wait complete. 2024-09-17T15:46:09.928 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:46:09.929 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-17T15:46:09.929 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-17T15:46:10.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.031+0000 7f6b353d3640 10 monclient: get_monmap_and_config 2024-09-17T15:46:10.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.031+0000 7f6b353d3640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.031+0000 7f6b353d3640 10 monclient: monmap: 2024-09-17T15:46:10.033 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:10.033 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.033 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:10.032149+0000 2024-09-17T15:46:10.033 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:10.032149+0000 2024-09-17T15:46:10.033 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.033 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.034 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:10.034 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:10.034 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:10.034 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:10.034 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:10.034 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:10.034 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:10.034 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:10.035 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:10.035 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.031+0000 7f6b353d3640 1 Processor -- start 2024-09-17T15:46:10.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.031+0000 7f6b353d3640 1 -- start start 2024-09-17T15:46:10.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.032+0000 7f6b353d3640 10 monclient: init 2024-09-17T15:46:10.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.032+0000 7f6b353d3640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.032+0000 7f6b353d3640 10 monclient: _add_conns ranks=[1,5,3,7,6,4,2,8,0] 2024-09-17T15:46:10.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.032+0000 7f6b353d3640 10 monclient(hunting): picked mon.noname-f con 0x7f6b300a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.032+0000 7f6b353d3640 10 monclient(hunting): picked mon.noname-h con 0x7f6b30135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.032+0000 7f6b353d3640 10 monclient(hunting): picked mon.noname-g con 0x7f6b30135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.032+0000 7f6b353d3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b3009d6f0 con 0x7f6b300a5520 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.032+0000 7f6b353d3640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b3009d9d0 con 0x7f6b30135f70 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.032+0000 7f6b353d3640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b3009dcb0 con 0x7f6b30135b50 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.032+0000 7f6b353d3640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.032+0000 7f6b353d3640 10 monclient(hunting): authenticate will time out at 767.707616s 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.032+0000 7f6b2e7fc640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7f6b30135b50 0x7f6b3013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:36810/0 (socket says 172.21.15.64:36810) 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.032+0000 7f6b2e7fc640 1 -- 172.21.15.64:0/3774787244 learned_addr learned my addr 172.21.15.64:0/3774787244 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 -- 172.21.15.64:0/3774787244 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 315464489 0 0) 0x7f6b3009d9d0 con 0x7f6b30135f70 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient(hunting): my global_id is 4476 2024-09-17T15:46:10.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 -- 172.21.15.64:0/3774787244 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b1c003520 con 0x7f6b30135f70 2024-09-17T15:46:10.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 -- 172.21.15.64:0/3774787244 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 616889800 0 0) 0x7f6b3009dcb0 con 0x7f6b30135b50 2024-09-17T15:46:10.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient(hunting): my global_id is 4406 2024-09-17T15:46:10.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 -- 172.21.15.64:0/3774787244 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b3009d9d0 con 0x7f6b30135b50 2024-09-17T15:46:10.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 -- 172.21.15.64:0/3774787244 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1881684089 0 0) 0x7f6b3009d6f0 con 0x7f6b300a5520 2024-09-17T15:46:10.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient(hunting): my global_id is 4501 2024-09-17T15:46:10.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 -- 172.21.15.64:0/3774787244 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b3009dcb0 con 0x7f6b300a5520 2024-09-17T15:46:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 -- 172.21.15.64:0/3774787244 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3286820419 0 0) 0x7f6b3009d9d0 con 0x7f6b30135b50 2024-09-17T15:46:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 -- 172.21.15.64:0/3774787244 >> v1:172.21.15.100:6790/0 conn(0x7f6b30135f70 legacy=0x7f6b301415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 -- 172.21.15.64:0/3774787244 >> v1:172.21.15.100:6789/0 conn(0x7f6b300a5520 legacy=0x7f6b300a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 monclient: found mon.noname-h 2024-09-17T15:46:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.100:6791/0 2024-09-17T15:46:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 -- 172.21.15.64:0/3774787244 --> v1:172.21.15.100:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b3009ccd0 con 0x7f6b30135b50 2024-09-17T15:46:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 -- 172.21.15.64:0/3774787244 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6b18001810 con 0x7f6b30135b50 2024-09-17T15:46:10.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient: dump: 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.047 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b353d3640 5 monclient: authenticate success, global_id 4406 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b353d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 -- 172.21.15.64:0/3774787244 <== mon.5 v1:172.21.15.100:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6b180019b0 con 0x7f6b30135b50 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 1 -- 172.21.15.64:0/3774787244 <== mon.5 v1:172.21.15.100:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6b18004be0 con 0x7f6b30135b50 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b2dffb640 10 monclient: dump: 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.048 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.049 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b353d3640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.033+0000 7f6b353d3640 4 set_mon_vals no callback set 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.034+0000 7f6b353d3640 10 monclient: shutdown 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.034+0000 7f6b353d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.034+0000 7f6b353d3640 1 -- 172.21.15.64:0/3774787244 >> v1:172.21.15.100:6791/0 conn(0x7f6b30135b50 legacy=0x7f6b3013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.034+0000 7f6b353d3640 1 -- 172.21.15.64:0/3774787244 shutdown_connections 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.034+0000 7f6b353d3640 1 -- 172.21.15.64:0/3774787244 >> 172.21.15.64:0/3774787244 conn(0x7f6b3006db30 msgr2=0x7f6b300ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.034+0000 7f6b353d3640 1 -- 172.21.15.64:0/3774787244 shutdown_connections 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.034+0000 7f6b353d3640 1 -- 172.21.15.64:0/3774787244 wait complete. 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.034+0000 7f6b353d3640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.034+0000 7f6b353d3640 10 monclient: monmap: 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:10.050 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.035+0000 7f6b353d3640 1 Processor -- start 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.035+0000 7f6b353d3640 1 -- start start 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.035+0000 7f6b353d3640 10 monclient: init 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.035+0000 7f6b353d3640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.035+0000 7f6b353d3640 10 monclient: _add_conns ranks=[7,3,8,0,1,2,5,4,6] 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.035+0000 7f6b353d3640 10 monclient(hunting): picked mon.noname-i con 0x7f6b300a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.035+0000 7f6b353d3640 10 monclient(hunting): picked mon.noname-g con 0x7f6b30135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.035+0000 7f6b353d3640 10 monclient(hunting): picked mon.noname-e con 0x7f6b30135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.035+0000 7f6b353d3640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b300a9250 con 0x7f6b30135b50 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.035+0000 7f6b353d3640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b300a2410 con 0x7f6b300a5520 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.035+0000 7f6b353d3640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b300a26f0 con 0x7f6b30135f70 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b353d3640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b353d3640 10 monclient(hunting): authenticate will time out at 767.711019s 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b2f7fe640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f6b30135f70 0x7f6b301d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38792/0 (socket says 172.21.15.64:38792) 2024-09-17T15:46:10.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b2f7fe640 1 -- 172.21.15.64:0/3646212966 learned_addr learned my addr 172.21.15.64:0/3646212966 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3779669917 0 0) 0x7f6b300a26f0 con 0x7f6b30135f70 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): my global_id is 4481 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b10003520 con 0x7f6b30135f70 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1424286175 0 0) 0x7f6b3009ccd0 con 0x7f6b30135b50 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): my global_id is 4485 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b300a26f0 con 0x7f6b30135b50 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2859825705 0 0) 0x7f6b300a2410 con 0x7f6b300a5520 2024-09-17T15:46:10.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): my global_id is 4399 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b3009ccd0 con 0x7f6b300a5520 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2927030264 0 0) 0x7f6b10003520 con 0x7f6b30135f70 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b300a2410 con 0x7f6b30135f70 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6b20001e00 con 0x7f6b30135f70 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): dump: 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.054 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:10.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2572558995 0 0) 0x7f6b300a26f0 con 0x7f6b30135b50 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b10003520 con 0x7f6b30135b50 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6b18001750 con 0x7f6b30135b50 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): dump: 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.056 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3174375419 0 0) 0x7f6b3009ccd0 con 0x7f6b300a5520 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b300a26f0 con 0x7f6b300a5520 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6b280012d0 con 0x7f6b300a5520 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient(hunting): dump: 2024-09-17T15:46:10.057 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.058 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2777057419 0 0) 0x7f6b300a2410 con 0x7f6b30135f70 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 >> v1:172.21.15.100:6792/0 conn(0x7f6b300a5520 legacy=0x7f6b3009e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 >> v1:172.21.15.100:6790/0 conn(0x7f6b30135b50 legacy=0x7f6b3009f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 monclient: found mon.e 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b30134f90 con 0x7f6b30135f70 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b353d3640 5 monclient: authenticate success, global_id 4481 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b353d3640 10 monclient: _renew_subs 2024-09-17T15:46:10.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b353d3640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:10.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b353d3640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6b300a0390 con 0x7f6b30135f70 2024-09-17T15:46:10.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b353d3640 10 monclient: _renew_subs 2024-09-17T15:46:10.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b353d3640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:46:10.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b353d3640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f6b300a1e50 con 0x7f6b30135f70 2024-09-17T15:46:10.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6b20003c20 con 0x7f6b30135f70 2024-09-17T15:46:10.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6b20004210 con 0x7f6b30135f70 2024-09-17T15:46:10.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:10.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b2d7fa640 4 set_mon_vals no callback set 2024-09-17T15:46:10.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.036+0000 7f6b277fe640 10 monclient: dump: 2024-09-17T15:46:10.060 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.060 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.061 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.037+0000 7f6b353d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.037+0000 7f6b353d3640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6b140065c0 con 0x7f6b14002860 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.037+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f6b20004430 con 0x7f6b30135f70 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (unknown 1990272338 0 0) 0x7f6b2004ee60 con 0x7f6b30135f70 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 405175112 0 0) 0x7f6b140065c0 con 0x7f6b14002860 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 10 monclient(hunting): my global_id is 4510 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b300a2410 con 0x7f6b14002860 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b2e7fc640 10 monclient: get_auth_request con 0x7f6b10040bc0 auth_method 0 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 141725188 0 0) 0x7f6b300a2410 con 0x7f6b14002860 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b140065c0 con 0x7f6b14002860 2024-09-17T15:46:10.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6b28003930 con 0x7f6b14002860 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 10 monclient: dump: 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.063 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.064 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.064 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.064 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.064 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3089322413 0 0) 0x7f6b140065c0 con 0x7f6b14002860 2024-09-17T15:46:10.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.038+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.100:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6b1400a160 con 0x7f6b14002860 2024-09-17T15:46:10.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.039+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6b28003d30 con 0x7f6b14002860 2024-09-17T15:46:10.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.039+0000 7f6b277fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.039+0000 7f6b277fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:10.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.039+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 >> v1:172.21.15.100:6789/0 conn(0x7f6b14002860 legacy=0x7f6b14004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.039+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 >> v1:172.21.15.100:6789/0 conn(0x7f6b14002860 legacy=0x7f6b14004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:10.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b353d3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:10.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b353d3640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6b14009e70 con 0x7f6b14009130 2024-09-17T15:46:10.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3366607833 0 0) 0x7f6b14009e70 con 0x7f6b14009130 2024-09-17T15:46:10.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b277fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b277fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b277fe640 10 monclient(hunting): my global_id is 4519 2024-09-17T15:46:10.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b140065c0 con 0x7f6b14009130 2024-09-17T15:46:10.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2609050953 0 0) 0x7f6b140065c0 con 0x7f6b14009130 2024-09-17T15:46:10.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b14009e70 con 0x7f6b14009130 2024-09-17T15:46:10.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6b280044d0 con 0x7f6b14009130 2024-09-17T15:46:10.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b277fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:10.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.045+0000 7f6b277fe640 10 monclient: dump: 2024-09-17T15:46:10.067 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.067 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.067 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.067 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.067 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.067 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.068 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.068 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.068 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.068 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.068 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.068 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.068 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.068 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.068 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.068 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.046+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2837672045 0 0) 0x7f6b14009e70 con 0x7f6b14009130 2024-09-17T15:46:10.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.046+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 --> v1:172.21.15.100:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6b1400c410 con 0x7f6b14009130 2024-09-17T15:46:10.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.046+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7710 (unknown 0 0 5666778) 0x7f6b28004730 con 0x7f6b14009130 2024-09-17T15:46:10.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.046+0000 7f6b277fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:10.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.046+0000 7f6b277fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:10.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.046+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 >> v1:172.21.15.100:6789/0 conn(0x7f6b14009130 legacy=0x7f6b140095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.046+0000 7f6b277fe640 1 -- 172.21.15.64:0/3646212966 >> v1:172.21.15.100:6789/0 conn(0x7f6b14009130 legacy=0x7f6b140095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:10.069 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:10.069 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:10.069 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:10.069 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:46:10.069 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 8, 2024-09-17T15:46:10.069 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 26697, 2024-09-17T15:46:10.070 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:10.070 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:10.070 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:10.070 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:10.070 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:46:10.070 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:10.070 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:46:10.070 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:46:10.070 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 26, 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:10.071 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:10.072 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:10.073 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:10.074 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:10.075 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:10.075 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:10.075 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:10.075 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:10.075 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:10.075 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.075 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:10.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.075 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:10.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.075 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:10.075 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:10.076 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.076 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.076 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:10.076 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.076 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.076 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:10.076 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:10.076 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.076 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.077 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.077 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:10.077 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:10.077 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.077 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.077 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.077 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:10.077 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.078 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.078 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:10.078 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:10.078 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.078 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.078 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.078 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:10.078 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:10.078 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.080 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.080 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:10.080 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.080 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.080 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:10.080 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:10.081 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.081 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.081 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.081 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:10.081 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:10.081 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.081 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.081 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.081 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:10.081 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.082 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.082 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:10.082 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:10.082 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.082 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.082 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.082 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:10.082 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:10.082 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.083 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.085 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:10.085 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:10.085 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.085 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.085 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.085 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:10.085 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:10.085 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.085 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.086 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:10.086 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.086 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.086 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.086 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:10.086 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:10.086 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.086 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.086 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.087 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:10.087 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:10.087 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.087 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.087 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.087 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:10.087 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.087 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.087 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.089 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.090 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.091 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:10.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.091 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.091 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:10.091 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:46:10.091 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.091 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.091 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:10.091 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:10.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.047+0000 7f6b353d3640 1 -- 172.21.15.64:0/3646212966 >> v1:172.21.15.100:6812/4210963484 conn(0x7f6b10040bc0 legacy=0x7f6b10043080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.047+0000 7f6b353d3640 10 monclient: shutdown 2024-09-17T15:46:10.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.047+0000 7f6b353d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:10.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.047+0000 7f6b353d3640 1 -- 172.21.15.64:0/3646212966 >> v1:172.21.15.64:6793/0 conn(0x7f6b30135f70 legacy=0x7f6b301d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.047+0000 7f6b2f7fe640 1 -- 172.21.15.64:0/3646212966 reap_dead start 2024-09-17T15:46:10.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.047+0000 7f6b2f7fe640 1 -- 172.21.15.64:0/3646212966 reap_dead start 2024-09-17T15:46:10.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.047+0000 7f6b353d3640 1 -- 172.21.15.64:0/3646212966 shutdown_connections 2024-09-17T15:46:10.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.047+0000 7f6b353d3640 1 -- 172.21.15.64:0/3646212966 >> 172.21.15.64:0/3646212966 conn(0x7f6b3006db30 msgr2=0x7f6b300a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:10.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.047+0000 7f6b353d3640 1 -- 172.21.15.64:0/3646212966 shutdown_connections 2024-09-17T15:46:10.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.047+0000 7f6b353d3640 1 -- 172.21.15.64:0/3646212966 wait complete. 2024-09-17T15:46:10.093 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:46:10.094 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-17T15:46:10.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-17T15:46:10.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.194+0000 7f82748ef640 10 monclient: get_monmap_and_config 2024-09-17T15:46:10.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.194+0000 7f82748ef640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.194+0000 7f82748ef640 10 monclient: monmap: 2024-09-17T15:46:10.196 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:10.196 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.196 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:10.194891+0000 2024-09-17T15:46:10.196 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:10.194891+0000 2024-09-17T15:46:10.196 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.194+0000 7f82748ef640 1 Processor -- start 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.194+0000 7f82748ef640 1 -- start start 2024-09-17T15:46:10.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.194+0000 7f82748ef640 10 monclient: init 2024-09-17T15:46:10.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f82748ef640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f82748ef640 10 monclient: _add_conns ranks=[7,0,5,6,4,3,2,8,1] 2024-09-17T15:46:10.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f82748ef640 10 monclient(hunting): picked mon.noname-i con 0x7f82700a54b0 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:10.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f82748ef640 10 monclient(hunting): picked mon.noname-a con 0x7f8270135c80 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:10.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f82748ef640 10 monclient(hunting): picked mon.noname-h con 0x7f82701360a0 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:10.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f82748ef640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82700aacb0 con 0x7f8270135c80 2024-09-17T15:46:10.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f82748ef640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f827009d780 con 0x7f82701360a0 2024-09-17T15:46:10.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f82748ef640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f827009da60 con 0x7f82700a54b0 2024-09-17T15:46:10.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f82748ef640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f82748ef640 10 monclient(hunting): authenticate will time out at 767.870490s 2024-09-17T15:46:10.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f826dd9b640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f8270135c80 0x7f827013df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59072/0 (socket says 172.21.15.64:59072) 2024-09-17T15:46:10.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f826dd9b640 1 -- 172.21.15.64:0/2189898272 learned_addr learned my addr 172.21.15.64:0/2189898272 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:10.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f826d59a640 1 -- 172.21.15.64:0/2189898272 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2788080278 0 0) 0x7f82700aacb0 con 0x7f8270135c80 2024-09-17T15:46:10.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f826d59a640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f826d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f826d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f826d59a640 10 monclient(hunting): my global_id is 4401 2024-09-17T15:46:10.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.195+0000 7f826d59a640 1 -- 172.21.15.64:0/2189898272 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8258003520 con 0x7f8270135c80 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 1 -- 172.21.15.64:0/2189898272 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3872109722 0 0) 0x7f827009da60 con 0x7f82700a54b0 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient(hunting): my global_id is 4408 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 1 -- 172.21.15.64:0/2189898272 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82700aacb0 con 0x7f82700a54b0 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 1 -- 172.21.15.64:0/2189898272 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2378448817 0 0) 0x7f827009d780 con 0x7f82701360a0 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient(hunting): my global_id is 4415 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 1 -- 172.21.15.64:0/2189898272 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f827009da60 con 0x7f82701360a0 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 1 -- 172.21.15.64:0/2189898272 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 255988288 0 0) 0x7f8258003520 con 0x7f8270135c80 2024-09-17T15:46:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 1 -- 172.21.15.64:0/2189898272 >> v1:172.21.15.100:6792/0 conn(0x7f82700a54b0 legacy=0x7f82700a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 1 -- 172.21.15.64:0/2189898272 >> v1:172.21.15.100:6791/0 conn(0x7f82701360a0 legacy=0x7f8270141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 1 monclient: found mon.noname-a 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 1 -- 172.21.15.64:0/2189898272 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827009dd40 con 0x7f8270135c80 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 1 -- 172.21.15.64:0/2189898272 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f82640017b0 con 0x7f8270135c80 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient: dump: 2024-09-17T15:46:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.208 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.208 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.208 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.208 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.208 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.208 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.208 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.208 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.208 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.208 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.208 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.208 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.208 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.209 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.209 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f82748ef640 5 monclient: authenticate success, global_id 4401 2024-09-17T15:46:10.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f82748ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:10.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 1 -- 172.21.15.64:0/2189898272 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8264001950 con 0x7f8270135c80 2024-09-17T15:46:10.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f82748ef640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:10.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f82748ef640 4 set_mon_vals no callback set 2024-09-17T15:46:10.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 1 -- 172.21.15.64:0/2189898272 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8264004be0 con 0x7f8270135c80 2024-09-17T15:46:10.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:10.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f826d59a640 10 monclient: dump: 2024-09-17T15:46:10.210 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.210 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.210 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.210 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.210 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.210 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.210 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.210 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.210 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.210 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.210 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f82748ef640 10 monclient: shutdown 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f82748ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f82748ef640 1 -- 172.21.15.64:0/2189898272 >> v1:172.21.15.64:6789/0 conn(0x7f8270135c80 legacy=0x7f827013df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f82748ef640 1 -- 172.21.15.64:0/2189898272 shutdown_connections 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f82748ef640 1 -- 172.21.15.64:0/2189898272 >> 172.21.15.64:0/2189898272 conn(0x7f827006daa0 msgr2=0x7f82700ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f82748ef640 1 -- 172.21.15.64:0/2189898272 shutdown_connections 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.196+0000 7f82748ef640 1 -- 172.21.15.64:0/2189898272 wait complete. 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.197+0000 7f82748ef640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.197+0000 7f82748ef640 10 monclient: monmap: 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:10.212 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.197+0000 7f82748ef640 1 Processor -- start 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.197+0000 7f82748ef640 1 -- start start 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.197+0000 7f82748ef640 10 monclient: init 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f82748ef640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f82748ef640 10 monclient: _add_conns ranks=[7,5,0,2,3,8,1,6,4] 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f82748ef640 10 monclient(hunting): picked mon.noname-i con 0x7f82700a54b0 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f82748ef640 10 monclient(hunting): picked mon.noname-h con 0x7f8270135c80 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f82748ef640 10 monclient(hunting): picked mon.noname-a con 0x7f82701360a0 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f82748ef640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82700a6c50 con 0x7f82701360a0 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f82748ef640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8270138900 con 0x7f8270135c80 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f82748ef640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8270137c80 con 0x7f82700a54b0 2024-09-17T15:46:10.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f82748ef640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f82748ef640 10 monclient(hunting): authenticate will time out at 767.873437s 2024-09-17T15:46:10.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f826ed9d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f82701360a0 0x7f82701d5e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59074/0 (socket says 172.21.15.64:59074) 2024-09-17T15:46:10.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f826ed9d640 1 -- 172.21.15.64:0/3989493056 learned_addr learned my addr 172.21.15.64:0/3989493056 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:10.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 675530473 0 0) 0x7f82700a6c50 con 0x7f82701360a0 2024-09-17T15:46:10.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f825effd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f825effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f825effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f825effd640 10 monclient(hunting): my global_id is 4410 2024-09-17T15:46:10.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8250003520 con 0x7f82701360a0 2024-09-17T15:46:10.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3948502340 0 0) 0x7f8250003520 con 0x7f82701360a0 2024-09-17T15:46:10.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82700a6c50 con 0x7f82701360a0 2024-09-17T15:46:10.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2167089557 0 0) 0x7f8270137c80 con 0x7f82700a54b0 2024-09-17T15:46:10.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f825effd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f825effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f825effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.198+0000 7f825effd640 10 monclient(hunting): my global_id is 4417 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8250003520 con 0x7f82700a54b0 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1508115591 0 0) 0x7f8270138900 con 0x7f8270135c80 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient(hunting): my global_id is 4424 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8270137c80 con 0x7f8270135c80 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8268004470 con 0x7f82701360a0 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient(hunting): dump: 2024-09-17T15:46:10.216 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.216 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.216 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.216 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.216 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.216 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.216 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.216 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.216 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.216 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.216 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.216 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.216 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2192150005 0 0) 0x7f82700a6c50 con 0x7f82701360a0 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 >> v1:172.21.15.100:6792/0 conn(0x7f82700a54b0 legacy=0x7f8270136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 >> v1:172.21.15.100:6791/0 conn(0x7f8270135c80 legacy=0x7f8270137570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 1 monclient: found mon.a 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827013a980 con 0x7f82701360a0 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f82748ef640 5 monclient: authenticate success, global_id 4410 2024-09-17T15:46:10.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f82748ef640 10 monclient: _renew_subs 2024-09-17T15:46:10.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f82748ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:10.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f82748ef640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f827013ac60 con 0x7f82701360a0 2024-09-17T15:46:10.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f82748ef640 10 monclient: _renew_subs 2024-09-17T15:46:10.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f82748ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:10.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f82748ef640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f82701424b0 con 0x7f82701360a0 2024-09-17T15:46:10.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f82680047f0 con 0x7f82701360a0 2024-09-17T15:46:10.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8268001840 con 0x7f82701360a0 2024-09-17T15:46:10.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f826cd99640 4 set_mon_vals no callback set 2024-09-17T15:46:10.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:10.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.199+0000 7f825effd640 10 monclient: dump: 2024-09-17T15:46:10.219 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.219 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.219 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.219 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.219 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.219 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.219 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.219 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.219 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f82748ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f82748ef640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f82540065c0 con 0x7f8254002860 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f82680019e0 con 0x7f82701360a0 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (unknown 1990272338 0 0) 0x7f826804eec0 con 0x7f82701360a0 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3234079486 0 0) 0x7f82540065c0 con 0x7f8254002860 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 10 monclient(hunting): my global_id is 4494 2024-09-17T15:46:10.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82700a6c50 con 0x7f8254002860 2024-09-17T15:46:10.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f826dd9b640 10 monclient: get_auth_request con 0x7f8250040540 auth_method 0 2024-09-17T15:46:10.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3464580154 0 0) 0x7f82700a6c50 con 0x7f8254002860 2024-09-17T15:46:10.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82540065c0 con 0x7f8254002860 2024-09-17T15:46:10.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8260001a20 con 0x7f8254002860 2024-09-17T15:46:10.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:46:10.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 10 monclient: dump: 2024-09-17T15:46:10.221 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.221 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.221 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.222 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.222 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.222 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.222 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.222 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.222 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.222 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.222 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.222 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.222 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.222 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.222 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.222 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 582477544 0 0) 0x7f82540065c0 con 0x7f8254002860 2024-09-17T15:46:10.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.200+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.100:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f825400a160 con 0x7f8254002860 2024-09-17T15:46:10.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.201+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f82600035f0 con 0x7f8254002860 2024-09-17T15:46:10.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.201+0000 7f825effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.201+0000 7f825effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:10.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.201+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 >> v1:172.21.15.100:6790/0 conn(0x7f8254002860 legacy=0x7f8254004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.201+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 >> v1:172.21.15.100:6790/0 conn(0x7f8254002860 legacy=0x7f8254004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:10.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.207+0000 7f82748ef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.207+0000 7f82748ef640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8254009e00 con 0x7f82540090e0 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 741091546 0 0) 0x7f8254009e00 con 0x7f82540090e0 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 10 monclient(hunting): my global_id is 4503 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82540065c0 con 0x7f82540090e0 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2272362755 0 0) 0x7f82540065c0 con 0x7f82540090e0 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8254009e00 con 0x7f82540090e0 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8260003cd0 con 0x7f82540090e0 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 10 monclient: dump: 2024-09-17T15:46:10.225 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.226 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.227 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1723126258 0 0) 0x7f8254009e00 con 0x7f82540090e0 2024-09-17T15:46:10.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 --> v1:172.21.15.100:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f825400c3a0 con 0x7f82540090e0 2024-09-17T15:46:10.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7710 (unknown 0 0 1420110829) 0x7f8260005e30 con 0x7f82540090e0 2024-09-17T15:46:10.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:10.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:10.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 >> v1:172.21.15.100:6790/0 conn(0x7f82540090e0 legacy=0x7f8254009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.208+0000 7f825effd640 1 -- 172.21.15.64:0/3989493056 >> v1:172.21.15.100:6790/0 conn(0x7f82540090e0 legacy=0x7f8254009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:10.227 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:10.227 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:10.228 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:10.228 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:46:10.228 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 8, 2024-09-17T15:46:10.228 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 26841, 2024-09-17T15:46:10.228 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:10.228 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:10.228 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:10.228 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:10.228 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:46:10.228 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:10.228 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:46:10.228 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:46:10.228 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 26, 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:10.229 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:10.230 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:10.231 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:10.232 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:10.232 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:10.232 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:10.232 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:10.232 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:10.232 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:10.232 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:10.232 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:10.232 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:10.232 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:10.232 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:10.232 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:10.232 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.234 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.234 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:10.234 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.234 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.234 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.234 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:10.234 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:10.234 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.234 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.234 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:10.235 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:10.236 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.236 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.236 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.236 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:10.236 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:10.236 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.236 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.236 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.236 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:10.236 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.237 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:10.237 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:10.237 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.237 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.237 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.237 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:10.237 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:10.238 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.238 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.238 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.238 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:10.238 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.238 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.238 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:10.238 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:10.238 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.238 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.239 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.239 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:10.239 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:10.239 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.239 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.239 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.239 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:10.239 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.240 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.240 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.240 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:10.240 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:10.240 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.240 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.240 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.240 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:10.240 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:10.240 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.242 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:10.242 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:10.242 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.242 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.242 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:10.242 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.242 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.242 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:10.243 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:10.243 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.243 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.243 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.243 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:10.243 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:10.243 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.243 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.243 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.243 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:10.244 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.245 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.245 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.245 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:10.245 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.245 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.245 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:10.245 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:10.245 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.245 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.245 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:10.246 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:46:10.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.247 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.247 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:10.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.247 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.247 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:10.247 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:46:10.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.247 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.247 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:10.247 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:10.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.209+0000 7f82748ef640 1 -- 172.21.15.64:0/3989493056 >> v1:172.21.15.100:6812/4210963484 conn(0x7f8250040540 legacy=0x7f8250042a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.209+0000 7f82748ef640 10 monclient: shutdown 2024-09-17T15:46:10.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.209+0000 7f82748ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:10.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.209+0000 7f82748ef640 1 -- 172.21.15.64:0/3989493056 >> v1:172.21.15.64:6789/0 conn(0x7f82701360a0 legacy=0x7f82701d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.209+0000 7f826ed9d640 1 -- 172.21.15.64:0/3989493056 reap_dead start 2024-09-17T15:46:10.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.209+0000 7f826ed9d640 1 -- 172.21.15.64:0/3989493056 reap_dead start 2024-09-17T15:46:10.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.209+0000 7f82748ef640 1 -- 172.21.15.64:0/3989493056 shutdown_connections 2024-09-17T15:46:10.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.209+0000 7f82748ef640 1 -- 172.21.15.64:0/3989493056 >> 172.21.15.64:0/3989493056 conn(0x7f827006daa0 msgr2=0x7f82700a4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:10.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.209+0000 7f82748ef640 1 -- 172.21.15.64:0/3989493056 shutdown_connections 2024-09-17T15:46:10.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.209+0000 7f82748ef640 1 -- 172.21.15.64:0/3989493056 wait complete. 2024-09-17T15:46:10.250 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:46:10.250 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-17T15:46:10.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-17T15:46:10.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.351+0000 7f6bbcd6b640 10 monclient: get_monmap_and_config 2024-09-17T15:46:10.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.351+0000 7f6bbcd6b640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.351+0000 7f6bbcd6b640 10 monclient: monmap: 2024-09-17T15:46:10.353 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:10.353 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.353 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:10.352393+0000 2024-09-17T15:46:10.353 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:10.352393+0000 2024-09-17T15:46:10.353 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.354 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.354 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:10.354 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:10.354 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:10.354 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:10.354 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:10.354 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:10.354 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:10.354 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:10.354 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:10.354 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bbcd6b640 1 Processor -- start 2024-09-17T15:46:10.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bbcd6b640 1 -- start start 2024-09-17T15:46:10.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bbcd6b640 10 monclient: init 2024-09-17T15:46:10.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bbcd6b640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bbcd6b640 10 monclient: _add_conns ranks=[2,3,5,7,6,8,0,4,1] 2024-09-17T15:46:10.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bbcd6b640 10 monclient(hunting): picked mon.noname-b con 0x7f6bb80a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:10.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bbcd6b640 10 monclient(hunting): picked mon.noname-g con 0x7f6bb8135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:10.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bbcd6b640 10 monclient(hunting): picked mon.noname-h con 0x7f6bb8135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:10.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bbcd6b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bb809d6f0 con 0x7f6bb80a5520 2024-09-17T15:46:10.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bbcd6b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bb809d9d0 con 0x7f6bb8135b50 2024-09-17T15:46:10.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bbcd6b640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bb809dcb0 con 0x7f6bb8135f70 2024-09-17T15:46:10.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bbcd6b640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bbcd6b640 10 monclient(hunting): authenticate will time out at 768.027935s 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.352+0000 7f6bb659c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f6bb80a5520 0x7f6bb80a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46922/0 (socket says 172.21.15.64:46922) 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb659c640 1 -- 172.21.15.64:0/4209604435 learned_addr learned my addr 172.21.15.64:0/4209604435 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 -- 172.21.15.64:0/4209604435 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 212875009 0 0) 0x7f6bb809d6f0 con 0x7f6bb80a5520 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient(hunting): my global_id is 4448 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 -- 172.21.15.64:0/4209604435 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ba0003520 con 0x7f6bb80a5520 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 -- 172.21.15.64:0/4209604435 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2930084371 0 0) 0x7f6bb809dcb0 con 0x7f6bb8135f70 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient(hunting): my global_id is 4433 2024-09-17T15:46:10.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 -- 172.21.15.64:0/4209604435 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bb809d6f0 con 0x7f6bb8135f70 2024-09-17T15:46:10.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 -- 172.21.15.64:0/4209604435 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3750924854 0 0) 0x7f6bb809d9d0 con 0x7f6bb8135b50 2024-09-17T15:46:10.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient(hunting): my global_id is 4512 2024-09-17T15:46:10.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 -- 172.21.15.64:0/4209604435 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bb809dcb0 con 0x7f6bb8135b50 2024-09-17T15:46:10.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 -- 172.21.15.64:0/4209604435 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3754058755 0 0) 0x7f6ba0003520 con 0x7f6bb80a5520 2024-09-17T15:46:10.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 -- 172.21.15.64:0/4209604435 >> v1:172.21.15.100:6791/0 conn(0x7f6bb8135f70 legacy=0x7f6bb81415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 -- 172.21.15.64:0/4209604435 >> v1:172.21.15.100:6790/0 conn(0x7f6bb8135b50 legacy=0x7f6bb813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 monclient: found mon.noname-b 2024-09-17T15:46:10.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:46:10.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 -- 172.21.15.64:0/4209604435 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bb809ccd0 con 0x7f6bb80a5520 2024-09-17T15:46:10.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bbcd6b640 5 monclient: authenticate success, global_id 4448 2024-09-17T15:46:10.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 -- 172.21.15.64:0/4209604435 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6ba8001830 con 0x7f6bb80a5520 2024-09-17T15:46:10.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:10.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient: dump: 2024-09-17T15:46:10.365 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.365 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.366 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bbcd6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:10.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 -- 172.21.15.64:0/4209604435 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ba8001a50 con 0x7f6bb80a5520 2024-09-17T15:46:10.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bbcd6b640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:10.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 1 -- 172.21.15.64:0/4209604435 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6ba8004be0 con 0x7f6bb80a5520 2024-09-17T15:46:10.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bbcd6b640 4 set_mon_vals no callback set 2024-09-17T15:46:10.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:10.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.353+0000 7f6bb559a640 10 monclient: dump: 2024-09-17T15:46:10.367 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.367 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.367 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.367 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.368 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.368 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.368 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.368 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.368 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.368 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.368 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.368 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.368 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.368 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.368 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.368 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.354+0000 7f6bbcd6b640 10 monclient: shutdown 2024-09-17T15:46:10.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.354+0000 7f6bbcd6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:10.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.354+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/4209604435 >> v1:172.21.15.64:6790/0 conn(0x7f6bb80a5520 legacy=0x7f6bb80a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.354+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/4209604435 shutdown_connections 2024-09-17T15:46:10.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.354+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/4209604435 >> 172.21.15.64:0/4209604435 conn(0x7f6bb806db30 msgr2=0x7f6bb80ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:10.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.354+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/4209604435 shutdown_connections 2024-09-17T15:46:10.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.354+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/4209604435 wait complete. 2024-09-17T15:46:10.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.354+0000 7f6bbcd6b640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 10 monclient: monmap: 2024-09-17T15:46:10.369 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:10.369 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:10.369 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:10.369 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:10.369 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 1 Processor -- start 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 1 -- start start 2024-09-17T15:46:10.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 10 monclient: init 2024-09-17T15:46:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 10 monclient: _add_conns ranks=[5,1,6,4,7,0,3,2,8] 2024-09-17T15:46:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 10 monclient(hunting): picked mon.noname-h con 0x7f6bb80a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 10 monclient(hunting): picked mon.noname-f con 0x7f6bb8135b50 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 10 monclient(hunting): picked mon.noname-d con 0x7f6bb8135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bb81419f0 con 0x7f6bb8135b50 2024-09-17T15:46:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bb813acf0 con 0x7f6bb80a5520 2024-09-17T15:46:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bb80a63e0 con 0x7f6bb8135f70 2024-09-17T15:46:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.355+0000 7f6bbcd6b640 10 monclient(hunting): authenticate will time out at 768.030953s 2024-09-17T15:46:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6bb659c640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7f6bb80a5520 0x7f6bb81370a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:36852/0 (socket says 172.21.15.64:36852) 2024-09-17T15:46:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6bb659c640 1 -- 172.21.15.64:0/2432193924 learned_addr learned my addr 172.21.15.64:0/2432193924 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3510207472 0 0) 0x7f6bb80a63e0 con 0x7f6bb8135f70 2024-09-17T15:46:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): my global_id is 4425 2024-09-17T15:46:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b98003520 con 0x7f6bb8135f70 2024-09-17T15:46:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2190124847 0 0) 0x7f6bb813acf0 con 0x7f6bb80a5520 2024-09-17T15:46:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): my global_id is 4442 2024-09-17T15:46:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bb80a63e0 con 0x7f6bb80a5520 2024-09-17T15:46:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3120787235 0 0) 0x7f6bb81419f0 con 0x7f6bb8135b50 2024-09-17T15:46:10.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): my global_id is 4528 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bb813acf0 con 0x7f6bb8135b50 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 689322053 0 0) 0x7f6b98003520 con 0x7f6bb8135f70 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bb81419f0 con 0x7f6bb8135f70 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6bb0001cb0 con 0x7f6bb8135f70 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): dump: 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.373 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:10.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1317130835 0 0) 0x7f6bb80a63e0 con 0x7f6bb80a5520 2024-09-17T15:46:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b98003520 con 0x7f6bb80a5520 2024-09-17T15:46:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6ba8004b20 con 0x7f6bb80a5520 2024-09-17T15:46:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:46:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): dump: 2024-09-17T15:46:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.376 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.376 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.376 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.376 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.376 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.376 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.376 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.376 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3464837317 0 0) 0x7f6bb813acf0 con 0x7f6bb8135b50 2024-09-17T15:46:10.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bb80a63e0 con 0x7f6bb8135b50 2024-09-17T15:46:10.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6bac0019d0 con 0x7f6bb8135b50 2024-09-17T15:46:10.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:10.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient(hunting): dump: 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.377 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.378 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.378 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.378 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4268909654 0 0) 0x7f6bb81419f0 con 0x7f6bb8135f70 2024-09-17T15:46:10.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 >> v1:172.21.15.100:6791/0 conn(0x7f6bb80a5520 legacy=0x7f6bb81370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 >> v1:172.21.15.100:6789/0 conn(0x7f6bb8135b50 legacy=0x7f6bb81377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 monclient: found mon.d 2024-09-17T15:46:10.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:10.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bb80a9030 con 0x7f6bb8135f70 2024-09-17T15:46:10.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6bbcd6b640 5 monclient: authenticate success, global_id 4425 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6bbcd6b640 10 monclient: _renew_subs 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6bbcd6b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6bb8137ec0 con 0x7f6bb8135f70 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6bbcd6b640 10 monclient: _renew_subs 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6bbcd6b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f6bb813a730 con 0x7f6bb8135f70 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6bb0003970 con 0x7f6bb8135f70 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6bb0003e60 con 0x7f6bb8135f70 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6bb4d99640 4 set_mon_vals no callback set 2024-09-17T15:46:10.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.356+0000 7f6ba6ffd640 10 monclient: dump: 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.381 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.381 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.381 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.357+0000 7f6bbcd6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.357+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6b9c0065c0 con 0x7f6b9c002860 2024-09-17T15:46:10.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.358+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f6bb0004000 con 0x7f6bb8135f70 2024-09-17T15:46:10.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.358+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(21..21 src has 1..21) ==== 3898+0+0 (unknown 1990272338 0 0) 0x7f6bb004dc50 con 0x7f6bb8135f70 2024-09-17T15:46:10.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.358+0000 7f6bb5d9b640 10 monclient: get_auth_request con 0x7f6b98040da0 auth_method 0 2024-09-17T15:46:10.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.365+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1239427085 0 0) 0x7f6b9c0065c0 con 0x7f6b9c002860 2024-09-17T15:46:10.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.365+0000 7f6ba6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.365+0000 7f6ba6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.365+0000 7f6ba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.365+0000 7f6ba6ffd640 10 monclient(hunting): my global_id is 4451 2024-09-17T15:46:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.366+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bb81419f0 con 0x7f6b9c002860 2024-09-17T15:46:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.376+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1441548622 0 0) 0x7f6bb81419f0 con 0x7f6b9c002860 2024-09-17T15:46:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.376+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b9c0065c0 con 0x7f6b9c002860 2024-09-17T15:46:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.376+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6ba8004ee0 con 0x7f6b9c002860 2024-09-17T15:46:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.376+0000 7f6ba6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.376+0000 7f6ba6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:46:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.376+0000 7f6ba6ffd640 10 monclient: dump: 2024-09-17T15:46:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.383 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.376+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3696470692 0 0) 0x7f6b9c0065c0 con 0x7f6b9c002860 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.376+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.100:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6b9c00a160 con 0x7f6b9c002860 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.377+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6ba8007480 con 0x7f6b9c002860 2024-09-17T15:46:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.377+0000 7f6ba6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.377+0000 7f6ba6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:10.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.377+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 >> v1:172.21.15.100:6791/0 conn(0x7f6b9c002860 legacy=0x7f6b9c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.377+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 >> v1:172.21.15.100:6791/0 conn(0x7f6b9c002860 legacy=0x7f6b9c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:10.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.386+0000 7f6bbcd6b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:10.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.386+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6b9c009e70 con 0x7f6b9c009130 2024-09-17T15:46:10.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3824590391 0 0) 0x7f6b9c009e70 con 0x7f6b9c009130 2024-09-17T15:46:10.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 10 monclient(hunting): my global_id is 4460 2024-09-17T15:46:10.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b9c0065c0 con 0x7f6b9c009130 2024-09-17T15:46:10.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3642110494 0 0) 0x7f6b9c0065c0 con 0x7f6b9c009130 2024-09-17T15:46:10.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b9c009e70 con 0x7f6b9c009130 2024-09-17T15:46:10.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6ba8005670 con 0x7f6b9c009130 2024-09-17T15:46:10.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:46:10.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 10 monclient: dump: 2024-09-17T15:46:10.390 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.390 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.390 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.390 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.390 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.390 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.390 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.391 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.391 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.391 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.391 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.391 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.391 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.391 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.391 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.391 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2955049142 0 0) 0x7f6b9c009e70 con 0x7f6b9c009130 2024-09-17T15:46:10.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.387+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 --> v1:172.21.15.100:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6b9c00c3f0 con 0x7f6b9c009130 2024-09-17T15:46:10.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 1461352276) 0x7f6ba8006b60 con 0x7f6b9c009130 2024-09-17T15:46:10.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6ba6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:10.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6ba6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:10.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 >> v1:172.21.15.100:6791/0 conn(0x7f6b9c009130 legacy=0x7f6b9c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6ba6ffd640 1 -- 172.21.15.64:0/2432193924 >> v1:172.21.15.100:6791/0 conn(0x7f6b9c009130 legacy=0x7f6b9c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:10.393 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:46:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 8, 2024-09-17T15:46:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 27040, 2024-09-17T15:46:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:46:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:46:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:46:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 26, 2024-09-17T15:46:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:46:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:10.418 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:10.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/2432193924 >> v1:172.21.15.100:6812/4210963484 conn(0x7f6b98040da0 legacy=0x7f6b98043260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6bbcd6b640 10 monclient: shutdown 2024-09-17T15:46:10.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6bbcd6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:10.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/2432193924 >> v1:172.21.15.64:6792/0 conn(0x7f6bb8135f70 legacy=0x7f6bb81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6bb6d9d640 1 -- 172.21.15.64:0/2432193924 reap_dead start 2024-09-17T15:46:10.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6bb6d9d640 1 -- 172.21.15.64:0/2432193924 reap_dead start 2024-09-17T15:46:10.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/2432193924 shutdown_connections 2024-09-17T15:46:10.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/2432193924 >> 172.21.15.64:0/2432193924 conn(0x7f6bb806db30 msgr2=0x7f6bb81340c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:10.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/2432193924 shutdown_connections 2024-09-17T15:46:10.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.388+0000 7f6bbcd6b640 1 -- 172.21.15.64:0/2432193924 wait complete. 2024-09-17T15:46:10.421 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:46:10.421 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-17T15:46:10.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-17T15:46:10.469 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[34971]: starting. 2024-09-17T15:46:10.469 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[34971]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-34970 2024-09-17T15:46:10.469 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[34971]: created object 0... 2024-09-17T15:46:10.469 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[34971]: created object 25... 2024-09-17T15:46:10.469 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[34971]: created object 49... 2024-09-17T15:46:10.469 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[34971]: finishing. 2024-09-17T15:46:10.469 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[34971]: shutting down. 2024-09-17T15:46:10.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.525+0000 7ff932008640 10 monclient: get_monmap_and_config 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.525+0000 7ff932008640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.525+0000 7ff932008640 10 monclient: monmap: 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:10.526062+0000 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:10.526062+0000 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:10.528 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:10.528 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:10.528 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:10.528 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.525+0000 7ff932008640 1 Processor -- start 2024-09-17T15:46:10.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff932008640 1 -- start start 2024-09-17T15:46:10.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff932008640 10 monclient: init 2024-09-17T15:46:10.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff932008640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff932008640 10 monclient: _add_conns ranks=[6,2,0,8,7,5,1,4,3] 2024-09-17T15:46:10.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff932008640 10 monclient(hunting): picked mon.noname-d con 0x7ff92c0a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:10.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff932008640 10 monclient(hunting): picked mon.noname-b con 0x7ff92c135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:10.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff932008640 10 monclient(hunting): picked mon.noname-a con 0x7ff92c135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:10.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff932008640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c09d6f0 con 0x7ff92c135f70 2024-09-17T15:46:10.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff932008640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c09d9d0 con 0x7ff92c135b50 2024-09-17T15:46:10.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff932008640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c09dcb0 con 0x7ff92c0a5520 2024-09-17T15:46:10.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff932008640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff932008640 10 monclient(hunting): authenticate will time out at 768.201564s 2024-09-17T15:46:10.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff92bfff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7ff92c135f70 0x7ff92c1415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59090/0 (socket says 172.21.15.64:59090) 2024-09-17T15:46:10.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff92b7fe640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7ff92c0a5520 0x7ff92c0a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:49622/0 (socket says 172.21.15.64:49622) 2024-09-17T15:46:10.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff92bfff640 1 -- 172.21.15.64:0/2516514654 learned_addr learned my addr 172.21.15.64:0/2516514654 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:10.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.526+0000 7ff92a7fc640 1 -- 172.21.15.64:0/2516514654 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1721547112 0 0) 0x7ff92c09dcb0 con 0x7ff92c0a5520 2024-09-17T15:46:10.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient(hunting): my global_id is 4434 2024-09-17T15:46:10.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 1 -- 172.21.15.64:0/2516514654 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff914003520 con 0x7ff92c0a5520 2024-09-17T15:46:10.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 1 -- 172.21.15.64:0/2516514654 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1679409539 0 0) 0x7ff92c09d6f0 con 0x7ff92c135f70 2024-09-17T15:46:10.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient(hunting): my global_id is 4419 2024-09-17T15:46:10.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 1 -- 172.21.15.64:0/2516514654 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff92c09dcb0 con 0x7ff92c135f70 2024-09-17T15:46:10.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 1 -- 172.21.15.64:0/2516514654 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3226006899 0 0) 0x7ff92c09d9d0 con 0x7ff92c135b50 2024-09-17T15:46:10.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient(hunting): my global_id is 4457 2024-09-17T15:46:10.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 1 -- 172.21.15.64:0/2516514654 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff92c09d6f0 con 0x7ff92c135b50 2024-09-17T15:46:10.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 1 -- 172.21.15.64:0/2516514654 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4054767696 0 0) 0x7ff914003520 con 0x7ff92c0a5520 2024-09-17T15:46:10.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 1 -- 172.21.15.64:0/2516514654 >> v1:172.21.15.64:6790/0 conn(0x7ff92c135b50 legacy=0x7ff92c13deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 1 -- 172.21.15.64:0/2516514654 >> v1:172.21.15.64:6789/0 conn(0x7ff92c135f70 legacy=0x7ff92c1415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 1 monclient: found mon.noname-d 2024-09-17T15:46:10.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:46:10.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 1 -- 172.21.15.64:0/2516514654 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c09ccd0 con 0x7ff92c0a5520 2024-09-17T15:46:10.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 1 -- 172.21.15.64:0/2516514654 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff91c001830 con 0x7ff92c0a5520 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient: dump: 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.540 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff932008640 5 monclient: authenticate success, global_id 4434 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff932008640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 1 -- 172.21.15.64:0/2516514654 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff91c001a50 con 0x7ff92c0a5520 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 1 -- 172.21.15.64:0/2516514654 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff91c001d40 con 0x7ff92c0a5520 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff932008640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:10.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff932008640 4 set_mon_vals no callback set 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff92a7fc640 10 monclient: dump: 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.542 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.543 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.543 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.543 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.543 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.543 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff932008640 10 monclient: shutdown 2024-09-17T15:46:10.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff932008640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:10.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff932008640 1 -- 172.21.15.64:0/2516514654 >> v1:172.21.15.64:6792/0 conn(0x7ff92c0a5520 legacy=0x7ff92c0a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.527+0000 7ff932008640 1 -- 172.21.15.64:0/2516514654 shutdown_connections 2024-09-17T15:46:10.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.528+0000 7ff932008640 1 -- 172.21.15.64:0/2516514654 >> 172.21.15.64:0/2516514654 conn(0x7ff92c06db30 msgr2=0x7ff92c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:10.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.528+0000 7ff932008640 1 -- 172.21.15.64:0/2516514654 shutdown_connections 2024-09-17T15:46:10.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.528+0000 7ff932008640 1 -- 172.21.15.64:0/2516514654 wait complete. 2024-09-17T15:46:10.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.528+0000 7ff932008640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.528+0000 7ff932008640 10 monclient: monmap: 2024-09-17T15:46:10.544 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:10.544 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:10.544 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:10.544 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:10.544 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.544 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:10.544 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:10.544 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:10.544 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:10.544 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:10.545 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:10.545 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:10.545 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:10.545 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:10.545 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:10.545 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.528+0000 7ff932008640 1 Processor -- start 2024-09-17T15:46:10.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.529+0000 7ff932008640 1 -- start start 2024-09-17T15:46:10.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.529+0000 7ff932008640 10 monclient: init 2024-09-17T15:46:10.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.529+0000 7ff932008640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.529+0000 7ff932008640 10 monclient: _add_conns ranks=[7,5,1,6,4,0,2,3,8] 2024-09-17T15:46:10.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.529+0000 7ff932008640 10 monclient(hunting): picked mon.noname-i con 0x7ff92c0a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:10.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.529+0000 7ff932008640 10 monclient(hunting): picked mon.noname-h con 0x7ff92c135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:10.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.529+0000 7ff932008640 10 monclient(hunting): picked mon.noname-f con 0x7ff92c135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:10.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.529+0000 7ff932008640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c0a8ce0 con 0x7ff92c135f70 2024-09-17T15:46:10.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.529+0000 7ff932008640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c134f40 con 0x7ff92c135b50 2024-09-17T15:46:10.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.529+0000 7ff932008640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c0a9230 con 0x7ff92c0a5520 2024-09-17T15:46:10.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.529+0000 7ff932008640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.529+0000 7ff932008640 10 monclient(hunting): authenticate will time out at 768.204718s 2024-09-17T15:46:10.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff92b7fe640 1 --1- >> v1:172.21.15.100:6792/0 conn(0x7ff92c0a5520 0x7ff92c09f0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6792/0 says I am v1:172.21.15.64:60586/0 (socket says 172.21.15.64:60586) 2024-09-17T15:46:10.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff92bfff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff92c135f70 0x7ff92c1d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.64:41344/0 (socket says 172.21.15.64:41344) 2024-09-17T15:46:10.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff92b7fe640 1 -- 172.21.15.64:0/1454655168 learned_addr learned my addr 172.21.15.64:0/1454655168 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:10.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1858897729 0 0) 0x7ff92c0a8ce0 con 0x7ff92c135f70 2024-09-17T15:46:10.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): my global_id is 4537 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff918003520 con 0x7ff92c135f70 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1171840591 0 0) 0x7ff92c134f40 con 0x7ff92c135b50 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): my global_id is 4469 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff92c0a8ce0 con 0x7ff92c135b50 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1740028631 0 0) 0x7ff92c0a9230 con 0x7ff92c0a5520 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): my global_id is 4426 2024-09-17T15:46:10.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff92c134f40 con 0x7ff92c0a5520 2024-09-17T15:46:10.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 398534088 0 0) 0x7ff918003520 con 0x7ff92c135f70 2024-09-17T15:46:10.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff92c0a9230 con 0x7ff92c135f70 2024-09-17T15:46:10.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3386235951 0 0) 0x7ff92c0a8ce0 con 0x7ff92c135b50 2024-09-17T15:46:10.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff918003520 con 0x7ff92c135b50 2024-09-17T15:46:10.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff924001c40 con 0x7ff92c135f70 2024-09-17T15:46:10.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:10.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:46:10.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): dump: 2024-09-17T15:46:10.548 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.548 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.548 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:10.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1153046419 0 0) 0x7ff92c134f40 con 0x7ff92c0a5520 2024-09-17T15:46:10.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff92c0a8ce0 con 0x7ff92c0a5520 2024-09-17T15:46:10.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff9200013a0 con 0x7ff92c135b50 2024-09-17T15:46:10.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:10.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:46:10.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): dump: 2024-09-17T15:46:10.550 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.550 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.550 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.550 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.550 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.550 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff91c0017f0 con 0x7ff92c0a5520 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:46:10.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient(hunting): dump: 2024-09-17T15:46:10.552 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.552 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.552 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.552 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.552 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.552 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.552 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.552 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.552 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.552 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.552 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.552 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 700155679 0 0) 0x7ff92c0a9230 con 0x7ff92c135f70 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 >> v1:172.21.15.100:6792/0 conn(0x7ff92c0a5520 legacy=0x7ff92c09f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 >> v1:172.21.15.100:6791/0 conn(0x7ff92c135b50 legacy=0x7ff92c09f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 monclient: found mon.f 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c0a0790 con 0x7ff92c135f70 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff913fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff932008640 5 monclient: authenticate success, global_id 4537 2024-09-17T15:46:10.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff932008640 10 monclient: _renew_subs 2024-09-17T15:46:10.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff932008640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:46:10.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff932008640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff92c0a2af0 con 0x7ff92c135f70 2024-09-17T15:46:10.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff932008640 10 monclient: _renew_subs 2024-09-17T15:46:10.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff932008640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:46:10.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.530+0000 7ff932008640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff92c09fed0 con 0x7ff92c135f70 2024-09-17T15:46:10.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.531+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.1 v1:172.21.15.100:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff924004b60 con 0x7ff92c135f70 2024-09-17T15:46:10.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.531+0000 7ff913fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.531+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.1 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff924005050 con 0x7ff92c135f70 2024-09-17T15:46:10.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.531+0000 7ff913fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.531+0000 7ff913fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.531+0000 7ff929ffb640 4 set_mon_vals no callback set 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.531+0000 7ff913fff640 10 monclient: dump: 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.555 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.556 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.556 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.556 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.556 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.556 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.531+0000 7ff932008640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.531+0000 7ff932008640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff90c0065c0 con 0x7ff90c002860 2024-09-17T15:46:10.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.531+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.1 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7ff924011930 con 0x7ff92c135f70 2024-09-17T15:46:10.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.1 v1:172.21.15.100:6789/0 8 ==== osd_map(22..22 src has 1..22) ==== 3898+0+0 (unknown 1212974561 0 0) 0x7ff92404ee20 con 0x7ff92c135f70 2024-09-17T15:46:10.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3783371951 0 0) 0x7ff90c0065c0 con 0x7ff90c002860 2024-09-17T15:46:10.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 10 monclient(hunting): my global_id is 4435 2024-09-17T15:46:10.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff92c0a9230 con 0x7ff90c002860 2024-09-17T15:46:10.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff92affd640 10 monclient: get_auth_request con 0x7ff918040c60 auth_method 0 2024-09-17T15:46:10.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2841835637 0 0) 0x7ff92c0a9230 con 0x7ff90c002860 2024-09-17T15:46:10.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff90c0065c0 con 0x7ff90c002860 2024-09-17T15:46:10.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff91c001c40 con 0x7ff90c002860 2024-09-17T15:46:10.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:46:10.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 10 monclient: dump: 2024-09-17T15:46:10.557 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.557 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.557 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.557 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.558 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.558 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.558 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.558 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.558 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.558 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.558 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.558 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.558 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.558 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.558 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.558 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1768176761 0 0) 0x7ff90c0065c0 con 0x7ff90c002860 2024-09-17T15:46:10.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.532+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff90c00a160 con 0x7ff90c002860 2024-09-17T15:46:10.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.533+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff91c007480 con 0x7ff90c002860 2024-09-17T15:46:10.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.533+0000 7ff913fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.533+0000 7ff913fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:10.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.533+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 >> v1:172.21.15.100:6792/0 conn(0x7ff90c002860 legacy=0x7ff90c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.533+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 >> v1:172.21.15.100:6792/0 conn(0x7ff90c002860 legacy=0x7ff90c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:10.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff932008640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff932008640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff90c009e70 con 0x7ff90c009130 2024-09-17T15:46:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3634381615 0 0) 0x7ff90c009e70 con 0x7ff90c009130 2024-09-17T15:46:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff913fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff913fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff913fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff913fff640 10 monclient(hunting): my global_id is 4444 2024-09-17T15:46:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff90c0065c0 con 0x7ff90c009130 2024-09-17T15:46:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2468648822 0 0) 0x7ff90c0065c0 con 0x7ff90c009130 2024-09-17T15:46:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff90c009e70 con 0x7ff90c009130 2024-09-17T15:46:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff91c005600 con 0x7ff90c009130 2024-09-17T15:46:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff913fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff913fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:46:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.539+0000 7ff913fff640 10 monclient: dump: 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.563 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3509088999 0 0) 0x7ff90c009e70 con 0x7ff90c009130 2024-09-17T15:46:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 --> v1:172.21.15.100:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff90c00c410 con 0x7ff90c009130 2024-09-17T15:46:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7710 (unknown 0 0 164185139) 0x7ff91c007630 con 0x7ff90c009130 2024-09-17T15:46:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff913fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff913fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 >> v1:172.21.15.100:6792/0 conn(0x7ff90c009130 legacy=0x7ff90c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff913fff640 1 -- 172.21.15.64:0/1454655168 >> v1:172.21.15.100:6792/0 conn(0x7ff90c009130 legacy=0x7ff90c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:10.563 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:10.564 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:10.564 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:10.564 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:46:10.564 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 8, 2024-09-17T15:46:10.564 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 27175, 2024-09-17T15:46:10.564 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:10.564 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:10.564 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:10.564 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:10.564 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:46:10.564 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:10.564 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:46:10.565 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:46:10.565 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:10.565 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:10.565 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.565 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 26, 2024-09-17T15:46:10.565 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:10.565 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:10.565 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:10.565 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:10.565 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:10.566 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:10.567 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:10.567 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:10.567 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:10.567 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:10.567 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:10.567 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:10.567 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:10.567 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:10.567 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:10.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.567 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:10.568 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:10.569 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:10.569 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:10.569 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:10.569 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:10.569 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:10.569 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:10.569 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:10.569 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:10.569 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:10.569 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:10.569 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:10.569 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:10.570 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.571 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.571 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:10.571 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:10.571 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.571 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.571 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.571 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:10.571 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.572 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.575 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.575 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:10.575 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:10.575 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.575 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.575 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.575 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:10.575 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:10.576 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.576 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.576 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:10.576 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.576 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:10.576 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:10.576 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.576 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.577 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.577 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:10.577 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:10.577 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.577 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.577 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:10.577 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.578 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:10.579 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.580 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.580 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.580 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:10.580 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.580 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.580 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:10.580 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:10.580 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.580 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.580 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.581 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:10.581 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:10.581 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:10.581 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:10.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:10.581 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:10.581 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:10.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.581 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:10.583 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:10.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.583 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.583 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:10.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.583 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.583 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:10.583 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:46:10.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.583 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:10.583 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": [ 2024-09-17T15:46:10.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:10.583 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.584 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:10.584 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:10.584 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:10.584 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:10.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:10.584 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:10.584 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:10.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff932008640 1 -- 172.21.15.64:0/1454655168 >> v1:172.21.15.100:6812/4210963484 conn(0x7ff918040c60 legacy=0x7ff918043120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff932008640 10 monclient: shutdown 2024-09-17T15:46:10.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff932008640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:10.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff932008640 1 -- 172.21.15.64:0/1454655168 >> v1:172.21.15.100:6789/0 conn(0x7ff92c135f70 legacy=0x7ff92c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff92bfff640 1 -- 172.21.15.64:0/1454655168 reap_dead start 2024-09-17T15:46:10.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff92bfff640 1 -- 172.21.15.64:0/1454655168 reap_dead start 2024-09-17T15:46:10.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff932008640 1 -- 172.21.15.64:0/1454655168 shutdown_connections 2024-09-17T15:46:10.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff932008640 1 -- 172.21.15.64:0/1454655168 >> 172.21.15.64:0/1454655168 conn(0x7ff92c06db30 msgr2=0x7ff92c0a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:10.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff932008640 1 -- 172.21.15.64:0/1454655168 shutdown_connections 2024-09-17T15:46:10.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.540+0000 7ff932008640 1 -- 172.21.15.64:0/1454655168 wait complete. 2024-09-17T15:46:10.586 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:46:10.587 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['g', 'd', 'h'] 2024-09-17T15:46:10.587 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:46:10.587 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-09-17T15:46:10.587 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-09-17T15:46:10.587 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-09-17T15:46:10.587 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:46:10.598 INFO:tasks.ceph.mon.g:Stopped 2024-09-17T15:46:10.599 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-09-17T15:46:10.599 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-09-17T15:46:10.599 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-09-17T15:46:10.599 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:46:10.610 INFO:tasks.ceph.mon.d:Stopped 2024-09-17T15:46:10.611 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h 2024-09-17T15:46:10.611 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-09-17T15:46:10.611 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-09-17T15:46:10.611 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:46:10.649 INFO:tasks.ceph.mon.h:Stopped 2024-09-17T15:46:10.650 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 6 2024-09-17T15:46:10.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:46:10.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.748+0000 7f965ee40640 10 monclient: get_monmap_and_config 2024-09-17T15:46:10.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.748+0000 7f965ee40640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.748+0000 7f965ee40640 10 monclient: monmap: 2024-09-17T15:46:10.760 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:10.760 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.760 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:10.749672+0000 2024-09-17T15:46:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:10.749672+0000 2024-09-17T15:46:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:10.762 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:10.762 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.749+0000 7f965ee40640 1 Processor -- start 2024-09-17T15:46:10.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.749+0000 7f965ee40640 1 -- start start 2024-09-17T15:46:10.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.749+0000 7f965ee40640 10 monclient: init 2024-09-17T15:46:10.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965ee40640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965ee40640 10 monclient: _add_conns ranks=[4,5,1,2,3,0,8,6,7] 2024-09-17T15:46:10.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965ee40640 10 monclient(hunting): picked mon.noname-c con 0x7f96580a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:10.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965ee40640 10 monclient(hunting): picked mon.noname-h con 0x7f9658135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:10.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965ee40640 10 monclient(hunting): picked mon.noname-f con 0x7f9658135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965ee40640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f965809d6f0 con 0x7f9658135f70 2024-09-17T15:46:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965ee40640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f965809d9d0 con 0x7f96580a5520 2024-09-17T15:46:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965ee40640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f965809dcb0 con 0x7f9658135b50 2024-09-17T15:46:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965ee40640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965ee40640 10 monclient(hunting): authenticate will time out at 768.425332s 2024-09-17T15:46:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965cbdc640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f96580a5520 0x7f96580a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:36942/0 (socket says 172.21.15.64:36942) 2024-09-17T15:46:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965cbdc640 1 -- 172.21.15.64:0/3639005668 learned_addr learned my addr 172.21.15.64:0/3639005668 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f9657fff640 1 -- 172.21.15.64:0/3639005668 >> v1:172.21.15.100:6791/0 conn(0x7f9658135b50 legacy=0x7f965813deb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 1 -- 172.21.15.64:0/3639005668 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 89751907 0 0) 0x7f965809d9d0 con 0x7f96580a5520 2024-09-17T15:46:10.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient(hunting): my global_id is 4558 2024-09-17T15:46:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 1 -- 172.21.15.64:0/3639005668 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9648003520 con 0x7f96580a5520 2024-09-17T15:46:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 1 -- 172.21.15.64:0/3639005668 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2684694349 0 0) 0x7f965809d6f0 con 0x7f9658135f70 2024-09-17T15:46:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient(hunting): my global_id is 4546 2024-09-17T15:46:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 1 -- 172.21.15.64:0/3639005668 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f965809d9d0 con 0x7f9658135f70 2024-09-17T15:46:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 1 -- 172.21.15.64:0/3639005668 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2864688416 0 0) 0x7f9648003520 con 0x7f96580a5520 2024-09-17T15:46:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 1 -- 172.21.15.64:0/3639005668 >> v1:172.21.15.100:6791/0 conn(0x7f9658135b50 legacy=0x7f965813deb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 1 -- 172.21.15.64:0/3639005668 >> v1:172.21.15.100:6789/0 conn(0x7f9658135f70 legacy=0x7f96581415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 1 monclient: found mon.noname-c 2024-09-17T15:46:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:46:10.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 1 -- 172.21.15.64:0/3639005668 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965809ccd0 con 0x7f96580a5520 2024-09-17T15:46:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 1 -- 172.21.15.64:0/3639005668 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f964c001830 con 0x7f96580a5520 2024-09-17T15:46:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965ee40640 5 monclient: authenticate success, global_id 4558 2024-09-17T15:46:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965ee40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient: dump: 2024-09-17T15:46:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.767 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.768 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.768 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f965ee40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 1 -- 172.21.15.64:0/3639005668 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f964c0019d0 con 0x7f96580a5520 2024-09-17T15:46:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 1 -- 172.21.15.64:0/3639005668 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f964c004be0 con 0x7f96580a5520 2024-09-17T15:46:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.750+0000 7f96577fe640 10 monclient: dump: 2024-09-17T15:46:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.769 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.770 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.770 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.770 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.770 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.770 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.770 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.770 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.770 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.771 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.771 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.771 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.771 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.751+0000 7f965ee40640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:10.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.751+0000 7f965ee40640 4 set_mon_vals no callback set 2024-09-17T15:46:10.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.751+0000 7f965ee40640 10 monclient: shutdown 2024-09-17T15:46:10.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.751+0000 7f965ee40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:10.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.751+0000 7f965ee40640 1 -- 172.21.15.64:0/3639005668 >> v1:172.21.15.64:6791/0 conn(0x7f96580a5520 legacy=0x7f96580a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.751+0000 7f965ee40640 1 -- 172.21.15.64:0/3639005668 shutdown_connections 2024-09-17T15:46:10.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.751+0000 7f965ee40640 1 -- 172.21.15.64:0/3639005668 >> 172.21.15.64:0/3639005668 conn(0x7f965806db30 msgr2=0x7f96580ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:10.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.751+0000 7f965ee40640 1 -- 172.21.15.64:0/3639005668 shutdown_connections 2024-09-17T15:46:10.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.751+0000 7f965ee40640 1 -- 172.21.15.64:0/3639005668 wait complete. 2024-09-17T15:46:10.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.752+0000 7f965ee40640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.752+0000 7f965ee40640 10 monclient: monmap: 2024-09-17T15:46:10.772 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:10.772 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:10.772 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:10.773 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:10.773 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.773 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:10.773 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:10.773 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:10.773 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:10.773 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:10.773 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:10.773 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:10.773 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:10.773 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:10.773 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:10.773 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.752+0000 7f965ee40640 1 Processor -- start 2024-09-17T15:46:10.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.752+0000 7f965ee40640 1 -- start start 2024-09-17T15:46:10.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.752+0000 7f965ee40640 10 monclient: init 2024-09-17T15:46:10.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965ee40640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965ee40640 10 monclient: _add_conns ranks=[0,8,1,3,7,4,5,2,6] 2024-09-17T15:46:10.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965ee40640 10 monclient(hunting): picked mon.noname-a con 0x7f96580a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:10.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965ee40640 10 monclient(hunting): picked mon.noname-e con 0x7f9658135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:10.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965ee40640 10 monclient(hunting): picked mon.noname-f con 0x7f9658135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:10.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965ee40640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96581352a0 con 0x7f96580a5520 2024-09-17T15:46:10.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965ee40640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96580a0860 con 0x7f9658135f70 2024-09-17T15:46:10.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965ee40640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9658134ee0 con 0x7f9658135b50 2024-09-17T15:46:10.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965ee40640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965ee40640 10 monclient(hunting): authenticate will time out at 768.428127s 2024-09-17T15:46:10.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965cbdc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f96580a5520 0x7f9658131ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59098/0 (socket says 172.21.15.64:59098) 2024-09-17T15:46:10.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f9657fff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f9658135b50 0x7f96581322b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38834/0 (socket says 172.21.15.64:38834) 2024-09-17T15:46:10.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965cbdc640 1 -- 172.21.15.64:0/2038938799 learned_addr learned my addr 172.21.15.64:0/2038938799 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:10.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3080689867 0 0) 0x7f9658134ee0 con 0x7f9658135b50 2024-09-17T15:46:10.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): my global_id is 4490 2024-09-17T15:46:10.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f963c003550 con 0x7f9658135b50 2024-09-17T15:46:10.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3392742078 0 0) 0x7f965809ccd0 con 0x7f96580a5520 2024-09-17T15:46:10.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): my global_id is 4428 2024-09-17T15:46:10.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9658134ee0 con 0x7f96580a5520 2024-09-17T15:46:10.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4130205597 0 0) 0x7f96580a0860 con 0x7f9658135f70 2024-09-17T15:46:10.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:10.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): my global_id is 4555 2024-09-17T15:46:10.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f965809ccd0 con 0x7f9658135f70 2024-09-17T15:46:10.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2790505826 0 0) 0x7f9658134ee0 con 0x7f96580a5520 2024-09-17T15:46:10.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96580a0860 con 0x7f96580a5520 2024-09-17T15:46:10.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f964c001770 con 0x7f96580a5520 2024-09-17T15:46:10.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:10.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:10.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): dump: 2024-09-17T15:46:10.777 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.777 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.777 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.777 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.777 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.777 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.777 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.778 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.778 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.778 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.778 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.778 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.778 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.778 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.778 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.778 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:10.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1312317117 0 0) 0x7f963c003550 con 0x7f9658135b50 2024-09-17T15:46:10.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9658134ee0 con 0x7f9658135b50 2024-09-17T15:46:10.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3730215724 0 0) 0x7f965809ccd0 con 0x7f9658135f70 2024-09-17T15:46:10.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f963c003550 con 0x7f9658135f70 2024-09-17T15:46:10.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f96440042e0 con 0x7f9658135b50 2024-09-17T15:46:10.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:10.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:10.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): dump: 2024-09-17T15:46:10.779 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.779 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.779 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.779 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.780 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.780 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.780 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.780 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.780 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.780 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.780 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.780 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.780 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.780 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.780 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.780 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9650001cf0 con 0x7f9658135f70 2024-09-17T15:46:10.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:10.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:10.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient(hunting): dump: 2024-09-17T15:46:10.781 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.781 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.781 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.781 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.781 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.781 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.781 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.781 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.781 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.781 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.782 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.782 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.782 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.782 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.782 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.782 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3159574374 0 0) 0x7f96580a0860 con 0x7f96580a5520 2024-09-17T15:46:10.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 >> v1:172.21.15.64:6793/0 conn(0x7f9658135b50 legacy=0x7f96581322b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 >> v1:172.21.15.100:6789/0 conn(0x7f9658135f70 legacy=0x7f96581d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 monclient: found mon.a 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965809e500 con 0x7f96580a5520 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f96557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965ee40640 5 monclient: authenticate success, global_id 4428 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965ee40640 10 monclient: _renew_subs 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.753+0000 7f965ee40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f965ee40640 1 -- 172.21.15.64:0/2038938799 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f965809f4a0 con 0x7f96580a5520 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f965ee40640 10 monclient: _renew_subs 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f965ee40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f965ee40640 1 -- 172.21.15.64:0/2038938799 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f96581329c0 con 0x7f96580a5520 2024-09-17T15:46:10.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f964c001990 con 0x7f96580a5520 2024-09-17T15:46:10.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f96557fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f964c004bd0 con 0x7f96580a5520 2024-09-17T15:46:10.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f96557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f96557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:10.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f9656ffd640 4 set_mon_vals no callback set 2024-09-17T15:46:10.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f96557fa640 10 monclient: dump: 2024-09-17T15:46:10.784 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:10.784 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:10.784 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:10.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f965ee40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f965ee40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f965ee40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:10.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f965ee40640 1 -- 172.21.15.64:0/2038938799 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9638005080 con 0x7f96580a5520 2024-09-17T15:46:10.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.754+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f964c011920 con 0x7f96580a5520 2024-09-17T15:46:10.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.755+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(22..22 src has 1..22) ==== 3898+0+0 (unknown 1212974561 0 0) 0x7f964c04ea60 con 0x7f96580a5520 2024-09-17T15:46:10.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.755+0000 7f9657fff640 10 monclient: get_auth_request con 0x7f963c040e30 auth_method 0 2024-09-17T15:46:10.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.759+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f96581352a0 con 0x7f96580a5520 2024-09-17T15:46:10.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.759+0000 7f96557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:10.759+0000 7f96557fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:11.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.015+0000 7f965ee40640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:46:11.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.015+0000 7f965ee40640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:11.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.015+0000 7f965ee40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:46:11.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.015+0000 7f965ee40640 1 -- 172.21.15.64:0/2038938799 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9638005370 con 0x7f96580a5520 2024-09-17T15:46:11.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.016+0000 7f96557fa640 1 -- 172.21.15.64:0/2038938799 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 1285061044) 0x7f964c0176f0 con 0x7f96580a5520 2024-09-17T15:46:11.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.016+0000 7f96557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:11.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.016+0000 7f96557fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:11.019 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":8,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:46:11.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.018+0000 7f965ee40640 1 -- 172.21.15.64:0/2038938799 >> v1:172.21.15.100:6812/4210963484 conn(0x7f963c040e30 legacy=0x7f963c0432f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:11.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.018+0000 7f965ee40640 10 monclient: shutdown 2024-09-17T15:46:11.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.018+0000 7f965ee40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:11.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.018+0000 7f965ee40640 1 -- 172.21.15.64:0/2038938799 >> v1:172.21.15.64:6789/0 conn(0x7f96580a5520 legacy=0x7f9658131ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:11.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.018+0000 7f965ee40640 1 -- 172.21.15.64:0/2038938799 shutdown_connections 2024-09-17T15:46:11.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.018+0000 7f965ee40640 1 -- 172.21.15.64:0/2038938799 >> 172.21.15.64:0/2038938799 conn(0x7f965806db30 msgr2=0x7f965812c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:11.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.018+0000 7f965ee40640 1 -- 172.21.15.64:0/2038938799 shutdown_connections 2024-09-17T15:46:11.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:11.018+0000 7f965ee40640 1 -- 172.21.15.64:0/2038938799 wait complete. 2024-09-17T15:46:14.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:46:14.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.142+0000 7f1803582640 10 monclient: get_monmap_and_config 2024-09-17T15:46:14.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.142+0000 7f1803582640 10 monclient: build_initial_monmap 2024-09-17T15:46:14.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.142+0000 7f1803582640 10 monclient: monmap: 2024-09-17T15:46:14.144 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:14.144 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:14.145 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:14.143421+0000 2024-09-17T15:46:14.145 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:14.143421+0000 2024-09-17T15:46:14.145 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:14.145 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:14.145 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:14.145 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:14.145 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:14.145 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:14.145 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:14.145 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:14.145 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:14.145 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:14.145 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:14.146 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:14.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.143+0000 7f1803582640 1 Processor -- start 2024-09-17T15:46:14.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.143+0000 7f1803582640 1 -- start start 2024-09-17T15:46:14.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.143+0000 7f1803582640 10 monclient: init 2024-09-17T15:46:14.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.143+0000 7f1803582640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:14.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.143+0000 7f1803582640 10 monclient: _add_conns ranks=[8,3,0,7,4,2,5,6,1] 2024-09-17T15:46:14.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.143+0000 7f1803582640 10 monclient(hunting): picked mon.noname-e con 0x7f17fc0a54b0 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:14.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.143+0000 7f1803582640 10 monclient(hunting): picked mon.noname-g con 0x7f17fc135c80 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:14.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.143+0000 7f1803582640 10 monclient(hunting): picked mon.noname-a con 0x7f17fc1360a0 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:14.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.143+0000 7f1803582640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17fc0aacb0 con 0x7f17fc1360a0 2024-09-17T15:46:14.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.143+0000 7f1803582640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17fc09d780 con 0x7f17fc135c80 2024-09-17T15:46:14.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.143+0000 7f1803582640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17fc09da60 con 0x7f17fc0a54b0 2024-09-17T15:46:14.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.143+0000 7f1803582640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:14.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.143+0000 7f1803582640 10 monclient(hunting): authenticate will time out at 771.818989s 2024-09-17T15:46:14.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f180131e640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f17fc0a54b0 0x7f17fc0a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38836/0 (socket says 172.21.15.64:38836) 2024-09-17T15:46:14.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f1801b1f640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f17fc1360a0 0x7f17fc141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59108/0 (socket says 172.21.15.64:59108) 2024-09-17T15:46:14.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f180131e640 1 -- 172.21.15.64:0/2618445833 learned_addr learned my addr 172.21.15.64:0/2618445833 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:14.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f1800b1d640 1 -- 172.21.15.64:0/2618445833 >> v1:172.21.15.100:6790/0 conn(0x7f17fc135c80 legacy=0x7f17fc13df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6790/0 2024-09-17T15:46:14.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 1 -- 172.21.15.64:0/2618445833 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4088410115 0 0) 0x7f17fc0aacb0 con 0x7f17fc1360a0 2024-09-17T15:46:14.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:14.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:14.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:14.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient(hunting): my global_id is 4437 2024-09-17T15:46:14.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 1 -- 172.21.15.64:0/2618445833 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17e8003520 con 0x7f17fc1360a0 2024-09-17T15:46:14.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 1 -- 172.21.15.64:0/2618445833 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2489536312 0 0) 0x7f17e8003520 con 0x7f17fc1360a0 2024-09-17T15:46:14.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 1 -- 172.21.15.64:0/2618445833 >> v1:172.21.15.64:6793/0 conn(0x7f17fc0a54b0 legacy=0x7f17fc0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:14.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 1 -- 172.21.15.64:0/2618445833 >> v1:172.21.15.100:6790/0 conn(0x7f17fc135c80 legacy=0x7f17fc13df10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:14.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:14.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 1 monclient: found mon.noname-a 2024-09-17T15:46:14.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:14.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:46:14.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 1 -- 172.21.15.64:0/2618445833 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17fc09dd40 con 0x7f17fc1360a0 2024-09-17T15:46:14.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient: _finish_auth 0 2024-09-17T15:46:14.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:14.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:14.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 1 -- 172.21.15.64:0/2618445833 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f17f4001ae0 con 0x7f17fc1360a0 2024-09-17T15:46:14.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:14.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:14.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient: dump: 2024-09-17T15:46:14.152 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:14.153 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:14.153 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:14.153 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:14.153 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:14.153 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:14.153 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:14.153 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:14.153 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:14.153 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:14.154 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:14.154 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:14.154 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:14.154 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:14.154 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:14.154 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:14.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f1803582640 5 monclient: authenticate success, global_id 4437 2024-09-17T15:46:14.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f1803582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:14.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 1 -- 172.21.15.64:0/2618445833 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17f4001d00 con 0x7f17fc1360a0 2024-09-17T15:46:14.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:14.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 1 -- 172.21.15.64:0/2618445833 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f17f4004ea0 con 0x7f17fc1360a0 2024-09-17T15:46:14.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:14.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f17f3fff640 10 monclient: dump: 2024-09-17T15:46:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:14.156 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:14.156 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:14.156 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:14.156 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:14.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f1803582640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:14.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.144+0000 7f1803582640 4 set_mon_vals no callback set 2024-09-17T15:46:14.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.145+0000 7f1803582640 10 monclient: shutdown 2024-09-17T15:46:14.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.145+0000 7f1803582640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:14.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.145+0000 7f1803582640 1 -- 172.21.15.64:0/2618445833 >> v1:172.21.15.64:6789/0 conn(0x7f17fc1360a0 legacy=0x7f17fc141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:14.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.145+0000 7f1803582640 1 -- 172.21.15.64:0/2618445833 shutdown_connections 2024-09-17T15:46:14.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.145+0000 7f1803582640 1 -- 172.21.15.64:0/2618445833 >> 172.21.15.64:0/2618445833 conn(0x7f17fc06daa0 msgr2=0x7f17fc0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:14.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.145+0000 7f1803582640 1 -- 172.21.15.64:0/2618445833 shutdown_connections 2024-09-17T15:46:14.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.145+0000 7f1803582640 1 -- 172.21.15.64:0/2618445833 wait complete. 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 10 monclient: build_initial_monmap 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 10 monclient: monmap: 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:14.157 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 1 Processor -- start 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 1 -- start start 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 10 monclient: init 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 10 monclient: _add_conns ranks=[4,1,8,0,2,6,5,7,3] 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 10 monclient(hunting): picked mon.noname-c con 0x7f17fc0a54b0 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 10 monclient(hunting): picked mon.noname-f con 0x7f17fc135c80 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 10 monclient(hunting): picked mon.noname-e con 0x7f17fc1360a0 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17fc132980 con 0x7f17fc135c80 2024-09-17T15:46:14.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17fc09f120 con 0x7f17fc0a54b0 2024-09-17T15:46:14.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17fc0a6fb0 con 0x7f17fc1360a0 2024-09-17T15:46:14.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:14.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f1803582640 10 monclient(hunting): authenticate will time out at 771.821918s 2024-09-17T15:46:14.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.146+0000 7f180131e640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f17fc0a54b0 0x7f17fc09e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:36954/0 (socket says 172.21.15.64:36954) 2024-09-17T15:46:14.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:14.147+0000 7f180131e640 1 -- 172.21.15.64:0/3828955887 learned_addr learned my addr 172.21.15.64:0/3828955887 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:17.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.146+0000 7f17f0ff9640 10 monclient(hunting): tick 2024-09-17T15:46:17.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:46:17.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 1 monclient(hunting): continuing hunt 2024-09-17T15:46:17.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:46:17.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6793/0 conn(0x7f17fc1360a0 legacy=0x7f17fc1d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:17.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6791/0 conn(0x7f17fc0a54b0 legacy=0x7f17fc09e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:17.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6789/0 conn(0x7f17fc135c80 legacy=0x7f17fc09ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:17.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 10 monclient: _add_conns ranks=[5,7,3,2,0,6] 2024-09-17T15:46:17.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 10 monclient(hunting): picked mon.noname-h con 0x7f17e40037e0 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:17.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 10 monclient(hunting): picked mon.noname-i con 0x7f17e4007450 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:17.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 10 monclient(hunting): picked mon.noname-g con 0x7f17e400b1a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:17.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17fc0a0e70 con 0x7f17e400b1a0 2024-09-17T15:46:17.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e400ffe0 con 0x7f17e40037e0 2024-09-17T15:46:17.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e4011280 con 0x7f17e4007450 2024-09-17T15:46:17.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f17f0ff9640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:17.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f180131e640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6791/0 conn(0x7f17e40037e0 legacy=0x7f17e4005c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:17.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.147+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6790/0 conn(0x7f17e400b1a0 legacy=0x7f17e400d660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6790/0 2024-09-17T15:46:17.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.348+0000 7f180131e640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6791/0 conn(0x7f17e40037e0 legacy=0x7f17e4005c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:17.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.348+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6790/0 conn(0x7f17e400b1a0 legacy=0x7f17e400d660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6790/0 2024-09-17T15:46:17.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.750+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6790/0 conn(0x7f17e400b1a0 legacy=0x7f17e400d660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6790/0 2024-09-17T15:46:17.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:17.750+0000 7f180131e640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6791/0 conn(0x7f17e40037e0 legacy=0x7f17e4005c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:18.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:18.550+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6790/0 conn(0x7f17e400b1a0 legacy=0x7f17e400d660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6790/0 2024-09-17T15:46:18.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:18.550+0000 7f180131e640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6791/0 conn(0x7f17e40037e0 legacy=0x7f17e4005c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:20.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 10 monclient(hunting): tick 2024-09-17T15:46:20.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:46:20.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 1 monclient(hunting): continuing hunt 2024-09-17T15:46:20.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:46:20.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6792/0 conn(0x7f17e4007450 legacy=0x7f17e40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6791/0 conn(0x7f17e40037e0 legacy=0x7f17e4005c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6790/0 conn(0x7f17e400b1a0 legacy=0x7f17e400d660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 10 monclient: _add_conns ranks=[0,2,6] 2024-09-17T15:46:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 reap_dead start 2024-09-17T15:46:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f17e4011670 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 reap_dead start 2024-09-17T15:46:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f17e400a0a0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 10 monclient(hunting): picked mon.noname-d con 0x7f17e400a500 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e4012580 con 0x7f17e4011670 2024-09-17T15:46:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17fc0a0e70 con 0x7f17e400a0a0 2024-09-17T15:46:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e400ffe0 con 0x7f17e400a500 2024-09-17T15:46:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f17f0ff9640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.146+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6792/0 conn(0x7f17e400a500 legacy=0x7f17e400a8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:46:20.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.348+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6792/0 conn(0x7f17e400a500 legacy=0x7f17e400a8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:46:20.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:20.749+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6792/0 conn(0x7f17e400a500 legacy=0x7f17e400a8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:46:21.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:21.551+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6792/0 conn(0x7f17e400a500 legacy=0x7f17e400a8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:46:23.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.146+0000 7f17f0ff9640 10 monclient(hunting): tick 2024-09-17T15:46:23.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.146+0000 7f17f0ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:46:23.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.146+0000 7f17f0ff9640 1 monclient(hunting): continuing hunt 2024-09-17T15:46:23.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.146+0000 7f17f0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:46:23.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.146+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6792/0 conn(0x7f17e400a500 legacy=0x7f17e400a8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:23.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.146+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6790/0 conn(0x7f17e400a0a0 legacy=0x7f17e4005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:23.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.146+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6789/0 conn(0x7f17e4011670 legacy=0x7f17e4011a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:23.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.146+0000 7f17f0ff9640 10 monclient: _add_conns ranks=[7,6,5,8,0,3,2,4,1] 2024-09-17T15:46:23.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.146+0000 7f17f0ff9640 10 monclient(hunting): picked mon.noname-i con 0x7f17e4009bd0 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:23.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.146+0000 7f17f0ff9640 10 monclient(hunting): picked mon.noname-d con 0x7f17e400a500 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:23.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.146+0000 7f17f0ff9640 10 monclient(hunting): picked mon.noname-h con 0x7f17e4010250 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:23.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.146+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e40098e0 con 0x7f17e4010250 2024-09-17T15:46:23.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.147+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e400ffe0 con 0x7f17e400a500 2024-09-17T15:46:23.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.147+0000 7f17f0ff9640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e4010f30 con 0x7f17e4009bd0 2024-09-17T15:46:23.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.147+0000 7f17f0ff9640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:23.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.147+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6791/0 conn(0x7f17e4010250 legacy=0x7f17e4010660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:23.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.147+0000 7f1800b1d640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6792/0 conn(0x7f17e400a500 legacy=0x7f17e400a8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:46:23.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.348+0000 7f1800b1d640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6792/0 conn(0x7f17e400a500 legacy=0x7f17e400a8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:46:23.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.348+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6791/0 conn(0x7f17e4010250 legacy=0x7f17e4010660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.749+0000 7f1800b1d640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6792/0 conn(0x7f17e400a500 legacy=0x7f17e400a8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:46:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:23.749+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6791/0 conn(0x7f17e4010250 legacy=0x7f17e4010660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:24.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:24.550+0000 7f1800b1d640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6792/0 conn(0x7f17e400a500 legacy=0x7f17e400a8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:46:24.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:24.550+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6791/0 conn(0x7f17e4010250 legacy=0x7f17e4010660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:25.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.383+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3055266351 0 0) 0x7f17e4010f30 con 0x7f17e4009bd0 2024-09-17T15:46:25.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.383+0000 7f17f1ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.383+0000 7f17f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.383+0000 7f17f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.383+0000 7f17f1ffb640 10 monclient(hunting): my global_id is 4453 2024-09-17T15:46:25.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.383+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17e0003520 con 0x7f17e4009bd0 2024-09-17T15:46:25.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 377294140 0 0) 0x7f17e0003520 con 0x7f17e4009bd0 2024-09-17T15:46:25.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17e4010f30 con 0x7f17e4009bd0 2024-09-17T15:46:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f17f8001b50 con 0x7f17e4009bd0 2024-09-17T15:46:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-17T15:46:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 10 monclient(hunting): dump: 2024-09-17T15:46:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.390 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.390 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.390 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.390 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.390 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.390 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.390 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:25.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2866938348 0 0) 0x7f17e4010f30 con 0x7f17e4009bd0 2024-09-17T15:46:25.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.64:6792/0 conn(0x7f17e400a500 legacy=0x7f17e400a8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:25.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6791/0 conn(0x7f17e4010250 legacy=0x7f17e4010660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:25.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:46:25.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 1 monclient: found mon.i 2024-09-17T15:46:25.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:25.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:46:25.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.100:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e401b920 con 0x7f17e4009bd0 2024-09-17T15:46:25.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:25.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:25.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:25.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f1803582640 5 monclient: authenticate success, global_id 4453 2024-09-17T15:46:25.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f1803582640 10 monclient: _renew_subs 2024-09-17T15:46:25.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f1803582640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:46:25.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f1803582640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.100:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f17fc09f670 con 0x7f17e4009bd0 2024-09-17T15:46:25.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 <== mon.7 v1:172.21.15.100:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17f8004a90 con 0x7f17e4009bd0 2024-09-17T15:46:25.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:25.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 <== mon.7 v1:172.21.15.100:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f17f8004f80 con 0x7f17e4009bd0 2024-09-17T15:46:25.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:25.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:46:25.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f1ffb640 10 monclient: dump: 2024-09-17T15:46:25.392 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.392 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.392 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.392 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.392 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.392 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.393 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.393 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.393 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.393 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.393 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.393 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.393 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.393 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.393 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.393 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.384+0000 7f17f37fe640 4 set_mon_vals no callback set 2024-09-17T15:46:25.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.385+0000 7f1803582640 10 monclient: _renew_subs 2024-09-17T15:46:25.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.385+0000 7f1803582640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:46:25.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.385+0000 7f1803582640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.100:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f17fc0a1870 con 0x7f17e4009bd0 2024-09-17T15:46:25.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.385+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 <== mon.7 v1:172.21.15.100:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f17f8011860 con 0x7f17e4009bd0 2024-09-17T15:46:25.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.386+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 <== mon.7 v1:172.21.15.100:6792/0 8 ==== osd_map(22..22 src has 1..22) ==== 3898+0+0 (unknown 1212974561 0 0) 0x7f17f804ed10 con 0x7f17e4009bd0 2024-09-17T15:46:25.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.386+0000 7f1800b1d640 10 monclient: get_auth_request con 0x7f17e0040160 auth_method 0 2024-09-17T15:46:25.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.386+0000 7f1803582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:25.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.386+0000 7f1803582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:25.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.386+0000 7f1803582640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:46:25.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.386+0000 7f1803582640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.100:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17fc0647b0 con 0x7f17e4009bd0 2024-09-17T15:46:25.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.392+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 <== mon.7 v1:172.21.15.100:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f17fc0647b0 con 0x7f17e4009bd0 2024-09-17T15:46:25.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.392+0000 7f17f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:25.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.392+0000 7f17f1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:25.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.652+0000 7f1803582640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:46:25.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.652+0000 7f1803582640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:25.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.652+0000 7f1803582640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:46:25.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.652+0000 7f1803582640 1 -- 172.21.15.64:0/3828955887 --> v1:172.21.15.100:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f17fc077070 con 0x7f17e4009bd0 2024-09-17T15:46:25.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.653+0000 7f17f1ffb640 1 -- 172.21.15.64:0/3828955887 <== mon.7 v1:172.21.15.100:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2711 (unknown 254746657 0 4123128949) 0x7f17fc077070 con 0x7f17e4009bd0 2024-09-17T15:46:25.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.653+0000 7f17f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:25.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.653+0000 7f17f1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:25.656 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":12,"quorum":[0,1,2,4,7,8],"quorum_names":["a","f","b","c","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:46:25.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.656+0000 7f1803582640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6812/4210963484 conn(0x7f17e0040160 legacy=0x7f17e0042620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.656+0000 7f1803582640 10 monclient: shutdown 2024-09-17T15:46:25.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.656+0000 7f1803582640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:25.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.656+0000 7f1803582640 1 -- 172.21.15.64:0/3828955887 >> v1:172.21.15.100:6792/0 conn(0x7f17e4009bd0 legacy=0x7f17e400d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.656+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 reap_dead start 2024-09-17T15:46:25.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.656+0000 7f1801b1f640 1 -- 172.21.15.64:0/3828955887 reap_dead start 2024-09-17T15:46:25.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.656+0000 7f1803582640 1 -- 172.21.15.64:0/3828955887 shutdown_connections 2024-09-17T15:46:25.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.656+0000 7f1803582640 1 -- 172.21.15.64:0/3828955887 >> 172.21.15.64:0/3828955887 conn(0x7f17fc06daa0 msgr2=0x7f17fc133db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:25.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.656+0000 7f1803582640 1 -- 172.21.15.64:0/3828955887 shutdown_connections 2024-09-17T15:46:25.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.656+0000 7f1803582640 1 -- 172.21.15.64:0/3828955887 wait complete. 2024-09-17T15:46:25.669 INFO:tasks.mon_thrash.ceph_manager:quorum is size 6 2024-09-17T15:46:25.669 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon a 2024-09-17T15:46:25.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:46:25.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.769+0000 7f758d040640 10 monclient: get_monmap_and_config 2024-09-17T15:46:25.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.769+0000 7f758d040640 10 monclient: build_initial_monmap 2024-09-17T15:46:25.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.769+0000 7f758d040640 10 monclient: monmap: 2024-09-17T15:46:25.772 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:25.772 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.772 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:25.771437+0000 2024-09-17T15:46:25.773 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:25.771437+0000 2024-09-17T15:46:25.773 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:25.773 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.773 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:25.773 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:25.773 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:25.773 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:25.773 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:25.773 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:25.773 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:25.773 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:25.773 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:25.773 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f758d040640 1 Processor -- start 2024-09-17T15:46:25.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f758d040640 1 -- start start 2024-09-17T15:46:25.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f758d040640 10 monclient: init 2024-09-17T15:46:25.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f758d040640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:25.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f758d040640 10 monclient: _add_conns ranks=[5,2,7,8,3,0,1,6,4] 2024-09-17T15:46:25.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f758d040640 10 monclient(hunting): picked mon.noname-h con 0x7f75880a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:25.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f758d040640 10 monclient(hunting): picked mon.noname-b con 0x7f7588135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:25.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f758d040640 10 monclient(hunting): picked mon.noname-i con 0x7f7588135f70 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:25.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f758d040640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f758809d6f0 con 0x7f7588135b50 2024-09-17T15:46:25.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f758d040640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f758809d9d0 con 0x7f75880a5520 2024-09-17T15:46:25.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f758d040640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f758809dcb0 con 0x7f7588135f70 2024-09-17T15:46:25.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f758d040640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:25.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f758d040640 10 monclient(hunting): authenticate will time out at 783.446881s 2024-09-17T15:46:25.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.770+0000 7f7586d9d640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7f75880a5520 legacy=0x7f75880a5900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:25.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f758659c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f7588135b50 0x7f758813deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35730/0 (socket says 172.21.15.64:35730) 2024-09-17T15:46:25.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f758659c640 1 -- 172.21.15.64:0/1542083196 learned_addr learned my addr 172.21.15.64:0/1542083196 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:25.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 1 -- 172.21.15.64:0/1542083196 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1888985668 0 0) 0x7f758809d6f0 con 0x7f7588135b50 2024-09-17T15:46:25.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient(hunting): my global_id is 4466 2024-09-17T15:46:25.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 1 -- 172.21.15.64:0/1542083196 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7570003520 con 0x7f7588135b50 2024-09-17T15:46:25.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 1 -- 172.21.15.64:0/1542083196 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1344364057 0 0) 0x7f758809dcb0 con 0x7f7588135f70 2024-09-17T15:46:25.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient(hunting): my global_id is 4462 2024-09-17T15:46:25.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 1 -- 172.21.15.64:0/1542083196 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f758809d6f0 con 0x7f7588135f70 2024-09-17T15:46:25.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 1 -- 172.21.15.64:0/1542083196 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2690630640 0 0) 0x7f7570003520 con 0x7f7588135b50 2024-09-17T15:46:25.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 1 -- 172.21.15.64:0/1542083196 >> v1:172.21.15.100:6792/0 conn(0x7f7588135f70 legacy=0x7f75881415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 1 -- 172.21.15.64:0/1542083196 >> v1:172.21.15.100:6791/0 conn(0x7f75880a5520 legacy=0x7f75880a5900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:25.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:46:25.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 1 monclient: found mon.noname-b 2024-09-17T15:46:25.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:25.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:46:25.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 1 -- 172.21.15.64:0/1542083196 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f758809ccd0 con 0x7f7588135b50 2024-09-17T15:46:25.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient: _finish_auth 0 2024-09-17T15:46:25.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:46:25.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:25.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 1 -- 172.21.15.64:0/1542083196 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f757c001830 con 0x7f7588135b50 2024-09-17T15:46:25.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f758d040640 5 monclient: authenticate success, global_id 4466 2024-09-17T15:46:25.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f758d040640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:25.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:25.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:25.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient: dump: 2024-09-17T15:46:25.788 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.788 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.788 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.788 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.788 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.789 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.789 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.789 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.789 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.789 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.789 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.789 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.789 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.790 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.790 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.790 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f758d040640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:25.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 1 -- 172.21.15.64:0/1542083196 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f757c001a50 con 0x7f7588135b50 2024-09-17T15:46:25.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:25.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 1 -- 172.21.15.64:0/1542083196 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f757c004be0 con 0x7f7588135b50 2024-09-17T15:46:25.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f758d040640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:25.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f758d040640 4 set_mon_vals no callback set 2024-09-17T15:46:25.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:25.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:25.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f7585d9b640 10 monclient: dump: 2024-09-17T15:46:25.791 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.791 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.791 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.792 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.792 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.792 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.792 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.792 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.792 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.792 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.792 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.792 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.792 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.792 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.793 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.793 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f758d040640 10 monclient: shutdown 2024-09-17T15:46:25.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f758d040640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:25.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.771+0000 7f758d040640 1 -- 172.21.15.64:0/1542083196 >> v1:172.21.15.64:6790/0 conn(0x7f7588135b50 legacy=0x7f758813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.772+0000 7f758d040640 1 -- 172.21.15.64:0/1542083196 shutdown_connections 2024-09-17T15:46:25.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.772+0000 7f758d040640 1 -- 172.21.15.64:0/1542083196 >> 172.21.15.64:0/1542083196 conn(0x7f758806db30 msgr2=0x7f75880ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:25.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.772+0000 7f758d040640 1 -- 172.21.15.64:0/1542083196 shutdown_connections 2024-09-17T15:46:25.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.772+0000 7f758d040640 1 -- 172.21.15.64:0/1542083196 wait complete. 2024-09-17T15:46:25.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.772+0000 7f758d040640 10 monclient: build_initial_monmap 2024-09-17T15:46:25.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.772+0000 7f758d040640 10 monclient: monmap: 2024-09-17T15:46:25.794 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:25.794 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:25.794 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:25.794 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:25.794 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:25.794 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:25.794 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:25.795 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:25.795 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:25.795 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:25.795 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:25.795 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:25.795 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:25.795 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:25.795 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:25.795 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758d040640 1 Processor -- start 2024-09-17T15:46:25.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758d040640 1 -- start start 2024-09-17T15:46:25.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758d040640 10 monclient: init 2024-09-17T15:46:25.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758d040640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758d040640 10 monclient: _add_conns ranks=[2,1,5,4,0,6,3,7,8] 2024-09-17T15:46:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758d040640 10 monclient(hunting): picked mon.noname-b con 0x7f75880a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758d040640 10 monclient(hunting): picked mon.noname-f con 0x7f7588135b50 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758d040640 10 monclient(hunting): picked mon.noname-h con 0x7f7588135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758d040640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f758809e8d0 con 0x7f7588135b50 2024-09-17T15:46:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758d040640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75881350a0 con 0x7f75880a5520 2024-09-17T15:46:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758d040640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75880a01d0 con 0x7f7588135f70 2024-09-17T15:46:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758d040640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758d040640 10 monclient(hunting): authenticate will time out at 783.449820s 2024-09-17T15:46:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f758759e640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7f7588135f70 legacy=0x7f75881d6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f7586d9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f75880a5520 0x7f7588131ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35744/0 (socket says 172.21.15.64:35744) 2024-09-17T15:46:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.773+0000 7f7586d9d640 1 -- 172.21.15.64:0/1784659289 learned_addr learned my addr 172.21.15.64:0/1784659289 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:25.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 8890977 0 0) 0x7f7578003a80 con 0x7f75880a5520 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): my global_id is 4475 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7574003550 con 0x7f75880a5520 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4135593185 0 0) 0x7f758809e8d0 con 0x7f7588135b50 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): my global_id is 4564 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7578003a80 con 0x7f7588135b50 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2981829082 0 0) 0x7f7574003550 con 0x7f75880a5520 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f758809e8d0 con 0x7f75880a5520 2024-09-17T15:46:25.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7578004570 con 0x7f75880a5520 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): dump: 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3601525467 0 0) 0x7f7578003a80 con 0x7f7588135b50 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7574003550 con 0x7f7588135b50 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f757c004b20 con 0x7f7588135b50 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient(hunting): dump: 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.801 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4204357531 0 0) 0x7f758809e8d0 con 0x7f75880a5520 2024-09-17T15:46:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 >> v1:172.21.15.100:6791/0 conn(0x7f7588135f70 legacy=0x7f75881d6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 >> v1:172.21.15.100:6789/0 conn(0x7f7588135b50 legacy=0x7f75881321d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 monclient: found mon.b 2024-09-17T15:46:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:25.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:25.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75880a04b0 con 0x7f75880a5520 2024-09-17T15:46:25.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:25.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:25.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:25.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f758d040640 5 monclient: authenticate success, global_id 4475 2024-09-17T15:46:25.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f758d040640 10 monclient: _renew_subs 2024-09-17T15:46:25.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f758d040640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:25.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f758d040640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f758809ef90 con 0x7f75880a5520 2024-09-17T15:46:25.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f758d040640 10 monclient: _renew_subs 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f758d040640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f758d040640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f758809fef0 con 0x7f75880a5520 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7578004710 con 0x7f75880a5520 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7578004d00 con 0x7f75880a5520 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f758559a640 4 set_mon_vals no callback set 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.774+0000 7f756f7fe640 10 monclient: dump: 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.803 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.804 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.804 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.804 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.804 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.804 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.804 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.804 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.804 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.804 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.804 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.804 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.775+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f7578004f20 con 0x7f75880a5520 2024-09-17T15:46:25.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.775+0000 7f758d040640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:25.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.775+0000 7f758d040640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f75680065c0 con 0x7f7568002860 2024-09-17T15:46:25.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.775+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(22..22 src has 1..22) ==== 3898+0+0 (unknown 1212974561 0 0) 0x7f757804ee80 con 0x7f75880a5520 2024-09-17T15:46:25.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.775+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418514989 0 0) 0x7f75680065c0 con 0x7f7568002860 2024-09-17T15:46:25.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 10 monclient(hunting): my global_id is 4446 2024-09-17T15:46:25.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f758809e8d0 con 0x7f7568002860 2024-09-17T15:46:25.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f758759e640 10 monclient: get_auth_request con 0x7f7574040850 auth_method 0 2024-09-17T15:46:25.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2180846440 0 0) 0x7f758809e8d0 con 0x7f7568002860 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f75680065c0 con 0x7f7568002860 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f757c001cd0 con 0x7f7568002860 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 10 monclient: dump: 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.806 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.807 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.807 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.807 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.807 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.807 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.807 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.807 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3965444628 0 0) 0x7f75680065c0 con 0x7f7568002860 2024-09-17T15:46:25.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.64:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f756800a160 con 0x7f7568002860 2024-09-17T15:46:25.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f757c005110 con 0x7f7568002860 2024-09-17T15:46:25.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:25.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:25.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 >> v1:172.21.15.64:6789/0 conn(0x7f7568002860 legacy=0x7f7568004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.776+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 >> v1:172.21.15.64:6789/0 conn(0x7f7568002860 legacy=0x7f7568004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:25.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.782+0000 7f758d040640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:25.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.782+0000 7f758d040640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7568009e70 con 0x7f7568009130 2024-09-17T15:46:25.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2590429588 0 0) 0x7f7568009e70 con 0x7f7568009130 2024-09-17T15:46:25.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 10 monclient(hunting): my global_id is 4455 2024-09-17T15:46:25.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75680065c0 con 0x7f7568009130 2024-09-17T15:46:25.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3963441118 0 0) 0x7f75680065c0 con 0x7f7568009130 2024-09-17T15:46:25.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7568009e70 con 0x7f7568009130 2024-09-17T15:46:25.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f757c0057f0 con 0x7f7568009130 2024-09-17T15:46:25.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:25.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:25.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 10 monclient: dump: 2024-09-17T15:46:25.811 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.811 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.811 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.811 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.811 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.811 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.811 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.811 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.811 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.811 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.811 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.812 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.812 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.812 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.812 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.812 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 607586059 0 0) 0x7f7568009e70 con 0x7f7568009130 2024-09-17T15:46:25.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 --> v1:172.21.15.64:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f756800c410 con 0x7f7568009130 2024-09-17T15:46:25.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7679 (unknown 0 0 3384671997) 0x7f757c005920 con 0x7f7568009130 2024-09-17T15:46:25.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:25.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:25.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 >> v1:172.21.15.64:6789/0 conn(0x7f7568009130 legacy=0x7f75680095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.783+0000 7f756f7fe640 1 -- 172.21.15.64:0/1784659289 >> v1:172.21.15.64:6789/0 conn(0x7f7568009130 legacy=0x7f75680095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:25.813 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "leader", 2024-09-17T15:46:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 12, 2024-09-17T15:46:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 42437, 2024-09-17T15:46:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.829 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.829 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:25.832 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:25.832 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:25.832 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:46:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:25.838 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:25.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.784+0000 7f758d040640 1 -- 172.21.15.64:0/1784659289 >> v1:172.21.15.100:6812/4210963484 conn(0x7f7574040850 legacy=0x7f7574042d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.784+0000 7f758d040640 10 monclient: shutdown 2024-09-17T15:46:25.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.784+0000 7f758d040640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:25.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.784+0000 7f758d040640 1 -- 172.21.15.64:0/1784659289 >> v1:172.21.15.64:6790/0 conn(0x7f75880a5520 legacy=0x7f7588131ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.784+0000 7f758759e640 1 -- 172.21.15.64:0/1784659289 reap_dead start 2024-09-17T15:46:25.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.784+0000 7f758759e640 1 -- 172.21.15.64:0/1784659289 reap_dead start 2024-09-17T15:46:25.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.784+0000 7f758d040640 1 -- 172.21.15.64:0/1784659289 shutdown_connections 2024-09-17T15:46:25.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.784+0000 7f758d040640 1 -- 172.21.15.64:0/1784659289 >> 172.21.15.64:0/1784659289 conn(0x7f758806db30 msgr2=0x7f75881334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:25.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.784+0000 7f758d040640 1 -- 172.21.15.64:0/1784659289 shutdown_connections 2024-09-17T15:46:25.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.784+0000 7f758d040640 1 -- 172.21.15.64:0/1784659289 wait complete. 2024-09-17T15:46:25.841 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-09-17T15:46:25.841 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon b 2024-09-17T15:46:25.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:46:25.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.942+0000 7f1a57c47640 10 monclient: get_monmap_and_config 2024-09-17T15:46:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.942+0000 7f1a57c47640 10 monclient: build_initial_monmap 2024-09-17T15:46:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.942+0000 7f1a57c47640 10 monclient: monmap: 2024-09-17T15:46:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:25.943968+0000 2024-09-17T15:46:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:25.943968+0000 2024-09-17T15:46:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:25.946 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:25.946 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:25.946 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:25.946 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:25.946 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:25.946 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:25.946 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.942+0000 7f1a57c47640 1 Processor -- start 2024-09-17T15:46:25.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.942+0000 7f1a57c47640 1 -- start start 2024-09-17T15:46:25.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a57c47640 10 monclient: init 2024-09-17T15:46:25.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a57c47640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:25.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a57c47640 10 monclient: _add_conns ranks=[6,0,7,3,8,4,5,2,1] 2024-09-17T15:46:25.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a57c47640 10 monclient(hunting): picked mon.noname-d con 0x7f1a500a54b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:25.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a57c47640 10 monclient(hunting): picked mon.noname-a con 0x7f1a50135c80 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:25.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a57c47640 10 monclient(hunting): picked mon.noname-i con 0x7f1a501360a0 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:25.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a559e3640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7f1a500a54b0 legacy=0x7f1a500a5890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:46:25.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a57c47640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a500aacb0 con 0x7f1a50135c80 2024-09-17T15:46:25.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a57c47640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a5009d780 con 0x7f1a500a54b0 2024-09-17T15:46:25.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a57c47640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a5009da60 con 0x7f1a501360a0 2024-09-17T15:46:25.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a57c47640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:25.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a57c47640 10 monclient(hunting): authenticate will time out at 783.619645s 2024-09-17T15:46:25.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a551e2640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f1a50135c80 0x7f1a5013df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:44024/0 (socket says 172.21.15.64:44024) 2024-09-17T15:46:25.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a551e2640 1 -- 172.21.15.64:0/3213032429 learned_addr learned my addr 172.21.15.64:0/3213032429 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:25.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a549e1640 1 -- 172.21.15.64:0/3213032429 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 256207264 0 0) 0x7f1a500aacb0 con 0x7f1a50135c80 2024-09-17T15:46:25.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a549e1640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a549e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a549e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a549e1640 10 monclient(hunting): my global_id is 4464 2024-09-17T15:46:25.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.943+0000 7f1a549e1640 1 -- 172.21.15.64:0/3213032429 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a3c003520 con 0x7f1a50135c80 2024-09-17T15:46:25.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 1 -- 172.21.15.64:0/3213032429 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4288228006 0 0) 0x7f1a5009da60 con 0x7f1a501360a0 2024-09-17T15:46:25.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient(hunting): my global_id is 4471 2024-09-17T15:46:25.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 1 -- 172.21.15.64:0/3213032429 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a500aacb0 con 0x7f1a501360a0 2024-09-17T15:46:25.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 1 -- 172.21.15.64:0/3213032429 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1012114272 0 0) 0x7f1a3c003520 con 0x7f1a50135c80 2024-09-17T15:46:25.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 1 -- 172.21.15.64:0/3213032429 >> v1:172.21.15.100:6792/0 conn(0x7f1a501360a0 legacy=0x7f1a50141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 1 -- 172.21.15.64:0/3213032429 >> v1:172.21.15.64:6792/0 conn(0x7f1a500a54b0 legacy=0x7f1a500a5890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:25.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient: _finish_hunting 0 2024-09-17T15:46:25.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 1 monclient: found mon.noname-a 2024-09-17T15:46:25.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:25.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:46:25.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 1 -- 172.21.15.64:0/3213032429 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a5009dd40 con 0x7f1a50135c80 2024-09-17T15:46:25.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient: _finish_auth 0 2024-09-17T15:46:25.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient: _check_auth_tickets 2024-09-17T15:46:25.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:25.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 1 -- 172.21.15.64:0/3213032429 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a40001830 con 0x7f1a50135c80 2024-09-17T15:46:25.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a57c47640 5 monclient: authenticate success, global_id 4464 2024-09-17T15:46:25.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a57c47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:25.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:25.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:25.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient: dump: 2024-09-17T15:46:25.959 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.959 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.959 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.959 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.959 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.959 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.959 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.959 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.959 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.959 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.960 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.960 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.960 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.960 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.960 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.960 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a57c47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:25.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 1 -- 172.21.15.64:0/3213032429 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a40001a50 con 0x7f1a50135c80 2024-09-17T15:46:25.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:25.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 1 -- 172.21.15.64:0/3213032429 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a40004be0 con 0x7f1a50135c80 2024-09-17T15:46:25.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a57c47640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:25.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a57c47640 4 set_mon_vals no callback set 2024-09-17T15:46:25.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:25.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:25.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a549e1640 10 monclient: dump: 2024-09-17T15:46:25.961 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.961 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.961 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.961 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.961 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.961 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.962 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.962 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.962 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.962 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.962 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.962 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.962 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.962 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.962 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.962 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a57c47640 10 monclient: shutdown 2024-09-17T15:46:25.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a57c47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:25.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a57c47640 1 -- 172.21.15.64:0/3213032429 >> v1:172.21.15.64:6789/0 conn(0x7f1a50135c80 legacy=0x7f1a5013df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a57c47640 1 -- 172.21.15.64:0/3213032429 shutdown_connections 2024-09-17T15:46:25.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a57c47640 1 -- 172.21.15.64:0/3213032429 >> 172.21.15.64:0/3213032429 conn(0x7f1a5006daa0 msgr2=0x7f1a500ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:25.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a57c47640 1 -- 172.21.15.64:0/3213032429 shutdown_connections 2024-09-17T15:46:25.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.944+0000 7f1a57c47640 1 -- 172.21.15.64:0/3213032429 wait complete. 2024-09-17T15:46:25.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.945+0000 7f1a57c47640 10 monclient: build_initial_monmap 2024-09-17T15:46:25.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.945+0000 7f1a57c47640 10 monclient: monmap: 2024-09-17T15:46:25.964 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:25.964 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:25.964 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:25.965 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:25.965 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:25.965 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:25.965 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:25.965 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:25.965 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:25.966 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:25.966 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:25.966 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:25.966 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:25.966 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:25.966 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:25.967 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.945+0000 7f1a57c47640 1 Processor -- start 2024-09-17T15:46:25.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.945+0000 7f1a57c47640 1 -- start start 2024-09-17T15:46:25.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.945+0000 7f1a57c47640 10 monclient: init 2024-09-17T15:46:25.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a57c47640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a57c47640 10 monclient: _add_conns ranks=[1,2,4,6,3,8,5,7,0] 2024-09-17T15:46:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a57c47640 10 monclient(hunting): picked mon.noname-f con 0x7f1a500a54b0 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a57c47640 10 monclient(hunting): picked mon.noname-b con 0x7f1a50135c80 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a57c47640 10 monclient(hunting): picked mon.noname-c con 0x7f1a501360a0 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:25.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a57c47640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a500a0170 con 0x7f1a500a54b0 2024-09-17T15:46:25.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a57c47640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a500a5110 con 0x7f1a50135c80 2024-09-17T15:46:25.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a57c47640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a500a7320 con 0x7f1a501360a0 2024-09-17T15:46:25.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a57c47640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:25.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a57c47640 10 monclient(hunting): authenticate will time out at 783.622451s 2024-09-17T15:46:25.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a551e2640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f1a50135c80 0x7f1a5009fa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35750/0 (socket says 172.21.15.64:35750) 2024-09-17T15:46:25.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a551e2640 1 -- 172.21.15.64:0/3587095174 learned_addr learned my addr 172.21.15.64:0/3587095174 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:25.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1303321967 0 0) 0x7f1a500a5110 con 0x7f1a50135c80 2024-09-17T15:46:25.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 10 monclient(hunting): my global_id is 4484 2024-09-17T15:46:25.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a34003520 con 0x7f1a50135c80 2024-09-17T15:46:25.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 128632052 0 0) 0x7f1a500a7320 con 0x7f1a501360a0 2024-09-17T15:46:25.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 10 monclient(hunting): my global_id is 4567 2024-09-17T15:46:25.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a500a5110 con 0x7f1a501360a0 2024-09-17T15:46:25.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4200848965 0 0) 0x7f1a500a0170 con 0x7f1a500a54b0 2024-09-17T15:46:25.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 10 monclient(hunting): my global_id is 4573 2024-09-17T15:46:25.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.946+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a500a7320 con 0x7f1a500a54b0 2024-09-17T15:46:25.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2695158134 0 0) 0x7f1a34003520 con 0x7f1a50135c80 2024-09-17T15:46:25.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a500a0170 con 0x7f1a50135c80 2024-09-17T15:46:25.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a40004b20 con 0x7f1a50135c80 2024-09-17T15:46:25.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:25.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:25.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient(hunting): dump: 2024-09-17T15:46:25.971 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.971 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.972 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.972 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.972 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.972 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.972 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.972 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.972 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.973 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.973 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.973 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.973 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.973 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.973 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.973 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:25.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 723290739 0 0) 0x7f1a500a5110 con 0x7f1a501360a0 2024-09-17T15:46:25.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a34003520 con 0x7f1a501360a0 2024-09-17T15:46:25.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a48001d30 con 0x7f1a501360a0 2024-09-17T15:46:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient(hunting): dump: 2024-09-17T15:46:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.975 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.975 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.975 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.975 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.975 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.975 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2521767681 0 0) 0x7f1a500a7320 con 0x7f1a500a54b0 2024-09-17T15:46:25.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a500a5110 con 0x7f1a500a54b0 2024-09-17T15:46:25.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a4c004260 con 0x7f1a500a54b0 2024-09-17T15:46:25.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:25.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:25.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient(hunting): dump: 2024-09-17T15:46:25.975 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.976 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.976 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.976 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.976 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.976 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.976 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.976 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.976 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.976 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.976 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.977 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.977 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.977 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.977 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.977 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3046469344 0 0) 0x7f1a500a0170 con 0x7f1a50135c80 2024-09-17T15:46:25.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 >> v1:172.21.15.64:6791/0 conn(0x7f1a501360a0 legacy=0x7f1a501d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 >> v1:172.21.15.100:6789/0 conn(0x7f1a500a54b0 legacy=0x7f1a5009f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:25.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 monclient: found mon.b 2024-09-17T15:46:25.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:25.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a500a1f60 con 0x7f1a50135c80 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a57c47640 5 monclient: authenticate success, global_id 4484 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a57c47640 10 monclient: _renew_subs 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a57c47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a57c47640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1a500a2520 con 0x7f1a50135c80 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a57c47640 10 monclient: _renew_subs 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a57c47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a57c47640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1a50132a80 con 0x7f1a50135c80 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a40004ea0 con 0x7f1a50135c80 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:25.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a40005390 con 0x7f1a50135c80 2024-09-17T15:46:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a47fff640 4 set_mon_vals no callback set 2024-09-17T15:46:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.947+0000 7f1a467fc640 10 monclient: dump: 2024-09-17T15:46:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.980 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.980 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.980 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.980 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.980 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.980 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.980 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.980 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.980 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.948+0000 7f1a57c47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:25.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.948+0000 7f1a57c47640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1a380065c0 con 0x7f1a38002860 2024-09-17T15:46:25.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.948+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f1a40011c70 con 0x7f1a50135c80 2024-09-17T15:46:25.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.948+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(22..22 src has 1..22) ==== 3898+0+0 (unknown 1212974561 0 0) 0x7f1a4004ebc0 con 0x7f1a50135c80 2024-09-17T15:46:25.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.948+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2218445565 0 0) 0x7f1a380065c0 con 0x7f1a38002860 2024-09-17T15:46:25.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.948+0000 7f1a467fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.948+0000 7f1a467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.948+0000 7f1a467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.948+0000 7f1a467fc640 10 monclient(hunting): my global_id is 4493 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.948+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a500a0170 con 0x7f1a38002860 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.948+0000 7f1a561e4640 10 monclient: get_auth_request con 0x7f1a34040c70 auth_method 0 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.949+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2751898561 0 0) 0x7f1a500a0170 con 0x7f1a38002860 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.949+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a380065c0 con 0x7f1a38002860 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.949+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a4c004a40 con 0x7f1a38002860 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.949+0000 7f1a467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.949+0000 7f1a467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.949+0000 7f1a467fc640 10 monclient: dump: 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.983 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.983 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.983 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.983 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.983 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.983 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.983 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.983 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.984 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.984 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.949+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1963105907 0 0) 0x7f1a380065c0 con 0x7f1a38002860 2024-09-17T15:46:25.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.949+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a3800a160 con 0x7f1a38002860 2024-09-17T15:46:25.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.949+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1a4c005fe0 con 0x7f1a38002860 2024-09-17T15:46:25.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.949+0000 7f1a467fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:25.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.949+0000 7f1a467fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:25.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.949+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 >> v1:172.21.15.64:6790/0 conn(0x7f1a38002860 legacy=0x7f1a38004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.949+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 >> v1:172.21.15.64:6790/0 conn(0x7f1a38002860 legacy=0x7f1a38004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:25.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a57c47640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:25.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a57c47640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1a38009e70 con 0x7f1a38009130 2024-09-17T15:46:25.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4149555566 0 0) 0x7f1a38009e70 con 0x7f1a38009130 2024-09-17T15:46:25.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:25.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 10 monclient(hunting): my global_id is 4502 2024-09-17T15:46:25.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a380065c0 con 0x7f1a38009130 2024-09-17T15:46:25.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1340191637 0 0) 0x7f1a380065c0 con 0x7f1a38009130 2024-09-17T15:46:25.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a38009e70 con 0x7f1a38009130 2024-09-17T15:46:25.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1a4c006800 con 0x7f1a38009130 2024-09-17T15:46:25.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:25.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:25.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 10 monclient: dump: 2024-09-17T15:46:25.989 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:25.989 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:25.989 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.989 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:25.989 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:25.990 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:25.990 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:25.990 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:25.990 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:25.990 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:25.990 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:25.991 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:25.991 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:25.991 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:25.991 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:25.991 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:25.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1234633609 0 0) 0x7f1a38009e70 con 0x7f1a38009130 2024-09-17T15:46:25.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 --> v1:172.21.15.64:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1a3800c410 con 0x7f1a38009130 2024-09-17T15:46:25.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7510 (unknown 0 0 1381026664) 0x7f1a4c0068f0 con 0x7f1a38009130 2024-09-17T15:46:25.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:25.992 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:25.992 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:25.992 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:25.992 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:46:25.992 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 12, 2024-09-17T15:46:25.992 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 42601, 2024-09-17T15:46:25.992 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:25.992 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:25.992 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:25.992 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:25.992 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:25.993 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:25.994 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:25.994 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:25.994 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:25.994 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:25.994 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:25.994 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:25.994 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:25.994 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:25.994 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:25.994 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:25.994 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:25.995 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:25.996 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:25.996 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:25.996 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:25.996 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:25.996 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:25.996 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:25.996 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:25.996 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:25.996 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:25.996 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:25.996 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:25.996 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:25.996 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:25.997 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:25.997 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:25.997 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:25.997 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:25.997 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:25.997 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:25.997 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:25.997 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:25.997 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:25.997 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:25.997 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:25.998 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:25.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.998 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:25.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.998 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:25.998 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:25.998 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:25.998 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:25.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:25.998 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:25.998 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:25.998 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:25.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:25.999 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:25.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:25.999 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:25.999 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:25.999 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:25.999 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:25.999 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.000 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:26.001 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:26.001 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.001 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.001 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.001 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:26.001 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:26.001 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.001 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.004 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:26.005 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.006 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.007 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:26.009 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.011 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:26.012 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:26.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:26.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 >> v1:172.21.15.64:6790/0 conn(0x7f1a38009130 legacy=0x7f1a380095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a467fc640 1 -- 172.21.15.64:0/3587095174 >> v1:172.21.15.64:6790/0 conn(0x7f1a38009130 legacy=0x7f1a380095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:26.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a57c47640 1 -- 172.21.15.64:0/3587095174 >> v1:172.21.15.100:6812/4210963484 conn(0x7f1a34040c70 legacy=0x7f1a34043130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a57c47640 10 monclient: shutdown 2024-09-17T15:46:26.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a57c47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:26.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a57c47640 1 -- 172.21.15.64:0/3587095174 >> v1:172.21.15.64:6790/0 conn(0x7f1a50135c80 legacy=0x7f1a5009fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.956+0000 7f1a561e4640 1 -- 172.21.15.64:0/3587095174 reap_dead start 2024-09-17T15:46:26.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.957+0000 7f1a561e4640 1 -- 172.21.15.64:0/3587095174 reap_dead start 2024-09-17T15:46:26.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.957+0000 7f1a57c47640 1 -- 172.21.15.64:0/3587095174 shutdown_connections 2024-09-17T15:46:26.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.957+0000 7f1a57c47640 1 -- 172.21.15.64:0/3587095174 >> 172.21.15.64:0/3587095174 conn(0x7f1a5006daa0 msgr2=0x7f1a500a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:26.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.957+0000 7f1a57c47640 1 -- 172.21.15.64:0/3587095174 shutdown_connections 2024-09-17T15:46:26.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:25.957+0000 7f1a57c47640 1 -- 172.21.15.64:0/3587095174 wait complete. 2024-09-17T15:46:26.015 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-09-17T15:46:26.015 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon c 2024-09-17T15:46:26.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:46:26.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.115+0000 7f509f117640 10 monclient: get_monmap_and_config 2024-09-17T15:46:26.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.115+0000 7f509f117640 10 monclient: build_initial_monmap 2024-09-17T15:46:26.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.115+0000 7f509f117640 10 monclient: monmap: 2024-09-17T15:46:26.118 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:26.118 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.118 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:26.117351+0000 2024-09-17T15:46:26.118 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:26.117351+0000 2024-09-17T15:46:26.118 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:26.118 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.118 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:26.119 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:26.119 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:26.119 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:26.119 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:26.119 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:26.119 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:26.119 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:26.119 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:26.119 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.116+0000 7f509f117640 1 Processor -- start 2024-09-17T15:46:26.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.116+0000 7f509f117640 1 -- start start 2024-09-17T15:46:26.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.116+0000 7f509f117640 10 monclient: init 2024-09-17T15:46:26.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.116+0000 7f509f117640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:26.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.116+0000 7f509f117640 10 monclient: _add_conns ranks=[8,5,3,0,6,2,7,1,4] 2024-09-17T15:46:26.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.116+0000 7f509f117640 10 monclient(hunting): picked mon.noname-e con 0x7f50980a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:26.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.116+0000 7f509f117640 10 monclient(hunting): picked mon.noname-h con 0x7f5098135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:26.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.116+0000 7f509f117640 10 monclient(hunting): picked mon.noname-g con 0x7f5098135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:46:26.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.116+0000 7f509f117640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f509809d6f0 con 0x7f5098135f70 2024-09-17T15:46:26.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.116+0000 7f509f117640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f509809d9d0 con 0x7f5098135b50 2024-09-17T15:46:26.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.116+0000 7f509f117640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f509809dcb0 con 0x7f50980a5520 2024-09-17T15:46:26.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.116+0000 7f509f117640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:26.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.116+0000 7f509f117640 10 monclient(hunting): authenticate will time out at 783.792993s 2024-09-17T15:46:26.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f509ceb3640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f50980a5520 0x7f50980a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51484/0 (socket says 172.21.15.64:51484) 2024-09-17T15:46:26.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f509ceb3640 1 -- 172.21.15.64:0/3174195677 learned_addr learned my addr 172.21.15.64:0/3174195677 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:26.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508ffff640 1 -- 172.21.15.64:0/3174195677 >> v1:172.21.15.100:6791/0 conn(0x7f5098135b50 legacy=0x7f509813deb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:26.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f509d6b4640 1 -- 172.21.15.64:0/3174195677 >> v1:172.21.15.100:6790/0 conn(0x7f5098135f70 legacy=0x7f50981415e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6790/0 2024-09-17T15:46:26.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 1 -- 172.21.15.64:0/3174195677 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 722408308 0 0) 0x7f509809dcb0 con 0x7f50980a5520 2024-09-17T15:46:26.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient(hunting): my global_id is 4499 2024-09-17T15:46:26.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 1 -- 172.21.15.64:0/3174195677 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5080003520 con 0x7f50980a5520 2024-09-17T15:46:26.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 1 -- 172.21.15.64:0/3174195677 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1115255695 0 0) 0x7f5080003520 con 0x7f50980a5520 2024-09-17T15:46:26.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 1 -- 172.21.15.64:0/3174195677 >> v1:172.21.15.100:6791/0 conn(0x7f5098135b50 legacy=0x7f509813deb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:26.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 1 -- 172.21.15.64:0/3174195677 >> v1:172.21.15.100:6790/0 conn(0x7f5098135f70 legacy=0x7f50981415e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:26.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:26.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 1 monclient: found mon.noname-e 2024-09-17T15:46:26.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:26.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:46:26.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 1 -- 172.21.15.64:0/3174195677 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509809ccd0 con 0x7f50980a5520 2024-09-17T15:46:26.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:26.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:26.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:26.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 1 -- 172.21.15.64:0/3174195677 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5094001830 con 0x7f50980a5520 2024-09-17T15:46:26.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f509f117640 5 monclient: authenticate success, global_id 4499 2024-09-17T15:46:26.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f509f117640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:26.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:46:26.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient: dump: 2024-09-17T15:46:26.132 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.132 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.132 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.132 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.132 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.132 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.132 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.132 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.133 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.133 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.133 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.133 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.133 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.133 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f509f117640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 1 -- 172.21.15.64:0/3174195677 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f50940019d0 con 0x7f50980a5520 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 1 -- 172.21.15.64:0/3174195677 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5094004be0 con 0x7f50980a5520 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f509f117640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f509f117640 4 set_mon_vals no callback set 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.117+0000 7f508f7fe640 10 monclient: dump: 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.134 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.118+0000 7f509f117640 10 monclient: shutdown 2024-09-17T15:46:26.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.118+0000 7f509f117640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:26.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.118+0000 7f509f117640 1 -- 172.21.15.64:0/3174195677 >> v1:172.21.15.64:6793/0 conn(0x7f50980a5520 legacy=0x7f50980a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.118+0000 7f509f117640 1 -- 172.21.15.64:0/3174195677 shutdown_connections 2024-09-17T15:46:26.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.118+0000 7f509f117640 1 -- 172.21.15.64:0/3174195677 >> 172.21.15.64:0/3174195677 conn(0x7f509806db30 msgr2=0x7f50980ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:26.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.118+0000 7f509f117640 1 -- 172.21.15.64:0/3174195677 shutdown_connections 2024-09-17T15:46:26.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.118+0000 7f509f117640 1 -- 172.21.15.64:0/3174195677 wait complete. 2024-09-17T15:46:26.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.118+0000 7f509f117640 10 monclient: build_initial_monmap 2024-09-17T15:46:26.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.118+0000 7f509f117640 10 monclient: monmap: 2024-09-17T15:46:26.136 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:26.136 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:26.136 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:26.137 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:26.137 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:26.137 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:26.137 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:26.137 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:26.137 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:26.137 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:26.137 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:26.137 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:26.137 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:26.138 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:26.138 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:26.138 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509f117640 1 Processor -- start 2024-09-17T15:46:26.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509f117640 1 -- start start 2024-09-17T15:46:26.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509f117640 10 monclient: init 2024-09-17T15:46:26.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509f117640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:26.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509f117640 10 monclient: _add_conns ranks=[6,2,5,4,3,8,7,0,1] 2024-09-17T15:46:26.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509f117640 10 monclient(hunting): picked mon.noname-d con 0x7f50980a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:26.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509f117640 10 monclient(hunting): picked mon.noname-b con 0x7f5098135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:26.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509f117640 10 monclient(hunting): picked mon.noname-h con 0x7f5098135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:26.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509ceb3640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7f50980a5520 legacy=0x7f50980a4ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:46:26.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509f117640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f509809f040 con 0x7f5098135b50 2024-09-17T15:46:26.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509f117640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50980a92c0 con 0x7f5098135f70 2024-09-17T15:46:26.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509f117640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50980a8d90 con 0x7f50980a5520 2024-09-17T15:46:26.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509f117640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:26.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509f117640 10 monclient(hunting): authenticate will time out at 783.795764s 2024-09-17T15:46:26.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f509d6b4640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7f5098135f70 legacy=0x7f50981d6170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:26.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f508ffff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f5098135b50 0x7f50981d2a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35770/0 (socket says 172.21.15.64:35770) 2024-09-17T15:46:26.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f508ffff640 1 -- 172.21.15.64:0/1923257996 learned_addr learned my addr 172.21.15.64:0/1923257996 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:26.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 946094939 0 0) 0x7f509809f040 con 0x7f5098135b50 2024-09-17T15:46:26.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f508d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f508d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f508d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f508d7fa640 10 monclient(hunting): my global_id is 4511 2024-09-17T15:46:26.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.119+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5084003520 con 0x7f5098135b50 2024-09-17T15:46:26.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 629968950 0 0) 0x7f5084003520 con 0x7f5098135b50 2024-09-17T15:46:26.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f509809f040 con 0x7f5098135b50 2024-09-17T15:46:26.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f50880043f0 con 0x7f5098135b50 2024-09-17T15:46:26.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:26.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:26.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 10 monclient(hunting): dump: 2024-09-17T15:46:26.141 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.141 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.141 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.141 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.141 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.141 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.141 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.141 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.142 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4061842546 0 0) 0x7f509809f040 con 0x7f5098135b50 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 >> v1:172.21.15.64:6792/0 conn(0x7f50980a5520 legacy=0x7f50980a4ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 >> v1:172.21.15.100:6791/0 conn(0x7f5098135f70 legacy=0x7f50981d6170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 10 monclient: _finish_hunting 0 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 1 monclient: found mon.b 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50980a0290 con 0x7f5098135b50 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f509f117640 5 monclient: authenticate success, global_id 4511 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f509f117640 10 monclient: _renew_subs 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f509f117640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:26.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f509f117640 1 -- 172.21.15.64:0/1923257996 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f50980a0570 con 0x7f5098135b50 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f509f117640 10 monclient: _renew_subs 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f509f117640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f509f117640 1 -- 172.21.15.64:0/1923257996 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f50980a51e0 con 0x7f5098135b50 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5088004610 con 0x7f5098135b50 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5088004bc0 con 0x7f5098135b50 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508effd640 4 set_mon_vals no callback set 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.120+0000 7f508d7fa640 10 monclient: dump: 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.144 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.145 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.145 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.145 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.145 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.145 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.146 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.146 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.146 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.146 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.146 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.146 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.146 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.121+0000 7f509f117640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:26.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.121+0000 7f509f117640 1 -- 172.21.15.64:0/1923257996 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f50780065c0 con 0x7f5078002860 2024-09-17T15:46:26.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.121+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f5088004e20 con 0x7f5098135b50 2024-09-17T15:46:26.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.121+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(22..22 src has 1..22) ==== 3898+0+0 (unknown 1212974561 0 0) 0x7f508804ed40 con 0x7f5098135b50 2024-09-17T15:46:26.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.121+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2794090734 0 0) 0x7f50780065c0 con 0x7f5078002860 2024-09-17T15:46:26.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.121+0000 7f508d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.121+0000 7f508d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.121+0000 7f508d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.121+0000 7f508d7fa640 10 monclient(hunting): my global_id is 4576 2024-09-17T15:46:26.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.121+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f509809f040 con 0x7f5078002860 2024-09-17T15:46:26.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.121+0000 7f509d6b4640 10 monclient: get_auth_request con 0x7f5084040150 auth_method 0 2024-09-17T15:46:26.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.121+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3559718856 0 0) 0x7f509809f040 con 0x7f5078002860 2024-09-17T15:46:26.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.122+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50780065c0 con 0x7f5078002860 2024-09-17T15:46:26.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.122+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5094004be0 con 0x7f5078002860 2024-09-17T15:46:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.122+0000 7f508d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.122+0000 7f508d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.122+0000 7f508d7fa640 10 monclient: dump: 2024-09-17T15:46:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.149 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.150 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.150 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.150 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.150 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.150 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.150 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.122+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 978943274 0 0) 0x7f50780065c0 con 0x7f5078002860 2024-09-17T15:46:26.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.122+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 --> v1:172.21.15.64:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f507800a160 con 0x7f5078002860 2024-09-17T15:46:26.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.122+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f50940014f0 con 0x7f5078002860 2024-09-17T15:46:26.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.122+0000 7f508d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:26.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.122+0000 7f508d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:26.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.122+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 >> v1:172.21.15.64:6791/0 conn(0x7f5078002860 legacy=0x7f5078004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.122+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 >> v1:172.21.15.64:6791/0 conn(0x7f5078002860 legacy=0x7f5078004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:26.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f509f117640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:26.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f509f117640 1 -- 172.21.15.64:0/1923257996 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5078009e50 con 0x7f5078009110 2024-09-17T15:46:26.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1879715671 0 0) 0x7f5078009e50 con 0x7f5078009110 2024-09-17T15:46:26.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 10 monclient(hunting): my global_id is 4585 2024-09-17T15:46:26.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50780065c0 con 0x7f5078009110 2024-09-17T15:46:26.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1213540740 0 0) 0x7f50780065c0 con 0x7f5078009110 2024-09-17T15:46:26.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5078009e50 con 0x7f5078009110 2024-09-17T15:46:26.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5094001890 con 0x7f5078009110 2024-09-17T15:46:26.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:26.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 10 monclient: dump: 2024-09-17T15:46:26.154 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.154 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.154 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.154 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.154 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.154 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.155 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.155 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.155 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.155 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.155 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.155 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.155 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.155 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.155 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.155 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 15546527 0 0) 0x7f5078009e50 con 0x7f5078009110 2024-09-17T15:46:26.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 --> v1:172.21.15.64:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f507800c3f0 con 0x7f5078009110 2024-09-17T15:46:26.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7510 (unknown 0 0 2071805478) 0x7f5094005340 con 0x7f5078009110 2024-09-17T15:46:26.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:26.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:26.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 >> v1:172.21.15.64:6791/0 conn(0x7f5078009110 legacy=0x7f50780095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f508d7fa640 1 -- 172.21.15.64:0/1923257996 >> v1:172.21.15.64:6791/0 conn(0x7f5078009110 legacy=0x7f50780095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:26.156 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:26.156 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:26.156 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 12, 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 42787, 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:26.157 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:26.158 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:26.158 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:26.158 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:26.158 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:26.158 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:26.158 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:26.158 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:26.158 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:26.158 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:26.158 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:26.158 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:26.158 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:26.158 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:26.159 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:26.161 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:26.161 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:26.161 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:26.161 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:26.161 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:26.161 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:26.161 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:26.161 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:26.161 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:26.161 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:26.162 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:26.162 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:26.162 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:26.162 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:26.162 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:26.162 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:26.162 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.163 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:26.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.163 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:26.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.163 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:26.163 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:26.164 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.164 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.164 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.164 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:26.164 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.165 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.165 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:26.165 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:26.165 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.165 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.165 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.166 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:26.166 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:26.166 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.166 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.167 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.167 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:26.167 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.167 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.168 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:26.168 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:26.168 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.168 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.168 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.169 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:26.169 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:26.169 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.169 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.169 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.169 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:26.169 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.170 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.171 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:26.171 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.171 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.171 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:26.171 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:26.171 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.172 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.172 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.172 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:26.172 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.174 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.175 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:26.176 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.177 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.178 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:26.178 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:26.178 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.178 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.179 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:26.179 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.179 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.179 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:26.179 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:26.180 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.180 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.180 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.180 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.181 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:26.181 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:26.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.181 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:26.181 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:26.181 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:26.181 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.181 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.181 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:26.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.181 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:26.182 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:26.182 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:46:26.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.182 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.182 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:26.182 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:26.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f509f117640 1 -- 172.21.15.64:0/1923257996 >> v1:172.21.15.100:6812/4210963484 conn(0x7f5084040150 legacy=0x7f5084042610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f509f117640 10 monclient: shutdown 2024-09-17T15:46:26.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f509f117640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:26.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f509f117640 1 -- 172.21.15.64:0/1923257996 >> v1:172.21.15.64:6790/0 conn(0x7f5098135b50 legacy=0x7f50981d2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.129+0000 7f509d6b4640 1 -- 172.21.15.64:0/1923257996 reap_dead start 2024-09-17T15:46:26.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.130+0000 7f509d6b4640 1 -- 172.21.15.64:0/1923257996 reap_dead start 2024-09-17T15:46:26.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.130+0000 7f509f117640 1 -- 172.21.15.64:0/1923257996 shutdown_connections 2024-09-17T15:46:26.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.130+0000 7f509f117640 1 -- 172.21.15.64:0/1923257996 >> 172.21.15.64:0/1923257996 conn(0x7f509806db30 msgr2=0x7f50980a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:26.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.130+0000 7f509f117640 1 -- 172.21.15.64:0/1923257996 shutdown_connections 2024-09-17T15:46:26.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.130+0000 7f509f117640 1 -- 172.21.15.64:0/1923257996 wait complete. 2024-09-17T15:46:26.185 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-09-17T15:46:26.185 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon e 2024-09-17T15:46:26.186 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-17T15:46:26.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 10 monclient: get_monmap_and_config 2024-09-17T15:46:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 10 monclient: build_initial_monmap 2024-09-17T15:46:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 10 monclient: monmap: 2024-09-17T15:46:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:26.287191+0000 2024-09-17T15:46:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:26.287191+0000 2024-09-17T15:46:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:26.289 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 1 Processor -- start 2024-09-17T15:46:26.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 1 -- start start 2024-09-17T15:46:26.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 10 monclient: init 2024-09-17T15:46:26.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 10 monclient: _add_conns ranks=[4,1,5,6,0,8,2,3,7] 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b80a54b0 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 10 monclient(hunting): picked mon.noname-f con 0x7fa8b8135c80 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 10 monclient(hunting): picked mon.noname-h con 0x7fa8b81360a0 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8b80aacb0 con 0x7fa8b8135c80 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8b809d780 con 0x7fa8b80a54b0 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8b809da60 con 0x7fa8b81360a0 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8c0573640 10 monclient(hunting): authenticate will time out at 783.962785s 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8be30f640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fa8b80a54b0 0x7fa8b80a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:47372/0 (socket says 172.21.15.64:47372) 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8be30f640 1 -- 172.21.15.64:0/2873732356 learned_addr learned my addr 172.21.15.64:0/2873732356 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.286+0000 7fa8beb10640 1 -- 172.21.15.64:0/2873732356 >> v1:172.21.15.100:6791/0 conn(0x7fa8b81360a0 legacy=0x7fa8b8141640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 1 -- 172.21.15.64:0/2873732356 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3285195449 0 0) 0x7fa8b809d780 con 0x7fa8b80a54b0 2024-09-17T15:46:26.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient(hunting): my global_id is 4594 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 1 -- 172.21.15.64:0/2873732356 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8a4003520 con 0x7fa8b80a54b0 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 1 -- 172.21.15.64:0/2873732356 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2352821820 0 0) 0x7fa8b80aacb0 con 0x7fa8b8135c80 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient(hunting): my global_id is 4582 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 1 -- 172.21.15.64:0/2873732356 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8b809d780 con 0x7fa8b8135c80 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 1 -- 172.21.15.64:0/2873732356 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3293768759 0 0) 0x7fa8a4003520 con 0x7fa8b80a54b0 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 1 -- 172.21.15.64:0/2873732356 >> v1:172.21.15.100:6791/0 conn(0x7fa8b81360a0 legacy=0x7fa8b8141640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 1 -- 172.21.15.64:0/2873732356 >> v1:172.21.15.100:6789/0 conn(0x7fa8b8135c80 legacy=0x7fa8b813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient: _finish_hunting 0 2024-09-17T15:46:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 1 monclient: found mon.noname-c 2024-09-17T15:46:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:46:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 1 -- 172.21.15.64:0/2873732356 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b809dd40 con 0x7fa8b80a54b0 2024-09-17T15:46:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient: _finish_auth 0 2024-09-17T15:46:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient: _check_auth_tickets 2024-09-17T15:46:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 1 -- 172.21.15.64:0/2873732356 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa8b4001830 con 0x7fa8b80a54b0 2024-09-17T15:46:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient: dump: 2024-09-17T15:46:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8c0573640 5 monclient: authenticate success, global_id 4594 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8c0573640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 1 -- 172.21.15.64:0/2873732356 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8b4001a50 con 0x7fa8b80a54b0 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 1 -- 172.21.15.64:0/2873732356 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa8b4004be0 con 0x7fa8b80a54b0 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8c0573640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8c0573640 4 set_mon_vals no callback set 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8bd30d640 10 monclient: dump: 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8c0573640 10 monclient: shutdown 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8c0573640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.287+0000 7fa8c0573640 1 -- 172.21.15.64:0/2873732356 >> v1:172.21.15.64:6791/0 conn(0x7fa8b80a54b0 legacy=0x7fa8b80a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.288+0000 7fa8c0573640 1 -- 172.21.15.64:0/2873732356 shutdown_connections 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.288+0000 7fa8c0573640 1 -- 172.21.15.64:0/2873732356 >> 172.21.15.64:0/2873732356 conn(0x7fa8b806daa0 msgr2=0x7fa8b80ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.288+0000 7fa8c0573640 1 -- 172.21.15.64:0/2873732356 shutdown_connections 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.288+0000 7fa8c0573640 1 -- 172.21.15.64:0/2873732356 wait complete. 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.288+0000 7fa8c0573640 10 monclient: build_initial_monmap 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.288+0000 7fa8c0573640 10 monclient: monmap: 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.288+0000 7fa8c0573640 1 Processor -- start 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8c0573640 1 -- start start 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8c0573640 10 monclient: init 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8c0573640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8c0573640 10 monclient: _add_conns ranks=[0,4,7,6,8,3,5,2,1] 2024-09-17T15:46:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8c0573640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b80a54b0 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8c0573640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b8135c80 addr v1:172.21.15.64:6791/0 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8c0573640 10 monclient(hunting): picked mon.noname-i con 0x7fa8b81360a0 addr v1:172.21.15.100:6792/0 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8c0573640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8b80a0c50 con 0x7fa8b80a54b0 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8c0573640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8b80a6fe0 con 0x7fa8b8135c80 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8be30f640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fa8b80a54b0 0x7fa8b8131900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:44040/0 (socket says 172.21.15.64:44040) 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8be30f640 1 -- 172.21.15.64:0/86784173 learned_addr learned my addr 172.21.15.64:0/86784173 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8bdb0e640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fa8b8135c80 0x7fa8b8132010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:47378/0 (socket says 172.21.15.64:47378) 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8c0573640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8b80a0830 con 0x7fa8b81360a0 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8c0573640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8c0573640 10 monclient(hunting): authenticate will time out at 783.965668s 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1502750359 0 0) 0x7fa8b809dd40 con 0x7fa8b80a54b0 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 10 monclient(hunting): my global_id is 4473 2024-09-17T15:46:26.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa89c003520 con 0x7fa8b80a54b0 2024-09-17T15:46:26.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4086562670 0 0) 0x7fa8b809d780 con 0x7fa8b8135c80 2024-09-17T15:46:26.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 10 monclient(hunting): my global_id is 4603 2024-09-17T15:46:26.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8b809dd40 con 0x7fa8b8135c80 2024-09-17T15:46:26.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 165613875 0 0) 0x7fa8b80a0830 con 0x7fa8b81360a0 2024-09-17T15:46:26.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 10 monclient(hunting): my global_id is 4480 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.289+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8b809d780 con 0x7fa8b81360a0 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 511975872 0 0) 0x7fa8b809dd40 con 0x7fa8b8135c80 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8b80a0830 con 0x7fa8b8135c80 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa8ac001a90 con 0x7fa8b8135c80 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient(hunting): dump: 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.308 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 280246083 0 0) 0x7fa89c003520 con 0x7fa8b80a54b0 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8b809dd40 con 0x7fa8b80a54b0 2024-09-17T15:46:26.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa8b4004b20 con 0x7fa8b80a54b0 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient(hunting): dump: 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.310 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3227311384 0 0) 0x7fa8b809d780 con 0x7fa8b81360a0 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa89c003520 con 0x7fa8b81360a0 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa8b0004440 con 0x7fa8b81360a0 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient(hunting): dump: 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.311 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3310561179 0 0) 0x7fa8b80a0830 con 0x7fa8b8135c80 2024-09-17T15:46:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 >> v1:172.21.15.100:6792/0 conn(0x7fa8b81360a0 legacy=0x7fa8b81d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 >> v1:172.21.15.64:6789/0 conn(0x7fa8b80a54b0 legacy=0x7fa8b8131900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 monclient: found mon.c 2024-09-17T15:46:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b80a69e0 con 0x7fa8b8135c80 2024-09-17T15:46:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient: _finish_auth 0 2024-09-17T15:46:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8c0573640 5 monclient: authenticate success, global_id 4603 2024-09-17T15:46:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8c0573640 10 monclient: _renew_subs 2024-09-17T15:46:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8c0573640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8c0573640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa8b809f9c0 con 0x7fa8b8135c80 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8c0573640 10 monclient: _renew_subs 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8c0573640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8c0573640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fa8b80a0550 con 0x7fa8b8135c80 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8ac0038c0 con 0x7fa8b8135c80 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa8ac003ef0 con 0x7fa8b8135c80 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8bcb0c640 4 set_mon_vals no callback set 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.290+0000 7fa8aaffd640 10 monclient: dump: 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.314 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.291+0000 7fa8c0573640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.291+0000 7fa8c0573640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa8a00065c0 con 0x7fa8a0002860 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.291+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fa8ac004090 con 0x7fa8b8135c80 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.291+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(22..22 src has 1..22) ==== 3898+0+0 (unknown 1212974561 0 0) 0x7fa8ac04eb70 con 0x7fa8b8135c80 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.291+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2328538325 0 0) 0x7fa8a00065c0 con 0x7fa8a0002860 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.291+0000 7fa8aaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.291+0000 7fa8aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.291+0000 7fa8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.291+0000 7fa8aaffd640 10 monclient(hunting): my global_id is 4508 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.291+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8b80a0830 con 0x7fa8a0002860 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.291+0000 7fa8beb10640 10 monclient: get_auth_request con 0x7fa89c040d40 auth_method 0 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.292+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2765841926 0 0) 0x7fa8b80a0830 con 0x7fa8a0002860 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.292+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8a00065c0 con 0x7fa8a0002860 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.292+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa8b40011f0 con 0x7fa8a0002860 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.292+0000 7fa8aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.292+0000 7fa8aaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.292+0000 7fa8aaffd640 10 monclient: dump: 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.317 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.318 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.318 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.318 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.292+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1900089666 0 0) 0x7fa8a00065c0 con 0x7fa8a0002860 2024-09-17T15:46:26.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.292+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa8a000a160 con 0x7fa8a0002860 2024-09-17T15:46:26.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.292+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa8b4005090 con 0x7fa8a0002860 2024-09-17T15:46:26.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.292+0000 7fa8aaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:26.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.292+0000 7fa8aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:26.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.292+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 >> v1:172.21.15.64:6793/0 conn(0x7fa8a0002860 legacy=0x7fa8a0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.292+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 >> v1:172.21.15.64:6793/0 conn(0x7fa8a0002860 legacy=0x7fa8a0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:26.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.298+0000 7fa8c0573640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.298+0000 7fa8c0573640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa8a0009e50 con 0x7fa8a0009110 2024-09-17T15:46:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.298+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1661422596 0 0) 0x7fa8a0009e50 con 0x7fa8a0009110 2024-09-17T15:46:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.298+0000 7fa8aaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.298+0000 7fa8aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.298+0000 7fa8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.298+0000 7fa8aaffd640 10 monclient(hunting): my global_id is 4517 2024-09-17T15:46:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.298+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8a00065c0 con 0x7fa8a0009110 2024-09-17T15:46:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 389193626 0 0) 0x7fa8a00065c0 con 0x7fa8a0009110 2024-09-17T15:46:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8a0009e50 con 0x7fa8a0009110 2024-09-17T15:46:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa8b4005770 con 0x7fa8a0009110 2024-09-17T15:46:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8aaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8aaffd640 10 monclient: dump: 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.321 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.322 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1987195713 0 0) 0x7fa8a0009e50 con 0x7fa8a0009110 2024-09-17T15:46:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 --> v1:172.21.15.64:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa8a000c3f0 con 0x7fa8a0009110 2024-09-17T15:46:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7677 (unknown 0 0 1029209995) 0x7fa8b40079b0 con 0x7fa8a0009110 2024-09-17T15:46:26.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8aaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:26.322 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:46:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 12, 2024-09-17T15:46:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 42947, 2024-09-17T15:46:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:46:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:46:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:26.341 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:26.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8aaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:26.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 >> v1:172.21.15.64:6793/0 conn(0x7fa8a0009110 legacy=0x7fa8a00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8aaffd640 1 -- 172.21.15.64:0/86784173 >> v1:172.21.15.64:6793/0 conn(0x7fa8a0009110 legacy=0x7fa8a00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:26.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8c0573640 1 -- 172.21.15.64:0/86784173 >> v1:172.21.15.100:6812/4210963484 conn(0x7fa89c040d40 legacy=0x7fa89c043200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8c0573640 10 monclient: shutdown 2024-09-17T15:46:26.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8c0573640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:26.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8c0573640 1 -- 172.21.15.64:0/86784173 >> v1:172.21.15.64:6791/0 conn(0x7fa8b8135c80 legacy=0x7fa8b8132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8beb10640 1 -- 172.21.15.64:0/86784173 reap_dead start 2024-09-17T15:46:26.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8beb10640 1 -- 172.21.15.64:0/86784173 reap_dead start 2024-09-17T15:46:26.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8c0573640 1 -- 172.21.15.64:0/86784173 shutdown_connections 2024-09-17T15:46:26.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8c0573640 1 -- 172.21.15.64:0/86784173 >> 172.21.15.64:0/86784173 conn(0x7fa8b806daa0 msgr2=0x7fa8b80a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:26.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8c0573640 1 -- 172.21.15.64:0/86784173 shutdown_connections 2024-09-17T15:46:26.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.299+0000 7fa8c0573640 1 -- 172.21.15.64:0/86784173 wait complete. 2024-09-17T15:46:26.344 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-09-17T15:46:26.344 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon f 2024-09-17T15:46:26.344 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-17T15:46:26.391 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[34972]: starting. 2024-09-17T15:46:26.392 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[34972]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-34970 2024-09-17T15:46:26.392 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[34972]: shutting down. 2024-09-17T15:46:26.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 10 monclient: get_monmap_and_config 2024-09-17T15:46:26.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 10 monclient: build_initial_monmap 2024-09-17T15:46:26.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 10 monclient: monmap: 2024-09-17T15:46:26.454 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:26.454 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.454 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:26.453079+0000 2024-09-17T15:46:26.454 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:26.453079+0000 2024-09-17T15:46:26.454 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:26.454 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.454 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:26.454 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:26.455 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:26.455 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:26.455 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:26.455 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:26.455 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:26.455 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:26.455 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:26.455 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 1 Processor -- start 2024-09-17T15:46:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 1 -- start start 2024-09-17T15:46:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 10 monclient: init 2024-09-17T15:46:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 10 monclient: _add_conns ranks=[2,6,8,5,0,7,3,4,1] 2024-09-17T15:46:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 10 monclient(hunting): picked mon.noname-b con 0x7f25500a54b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 10 monclient(hunting): picked mon.noname-d con 0x7f2550135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 10 monclient(hunting): picked mon.noname-e con 0x7f25501360a0 addr v1:172.21.15.64:6793/0 2024-09-17T15:46:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25500aacb0 con 0x7f25500a54b0 2024-09-17T15:46:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f255009d780 con 0x7f2550135c80 2024-09-17T15:46:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f255009da60 con 0x7f25501360a0 2024-09-17T15:46:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f2555c39640 10 monclient(hunting): authenticate will time out at 784.128586s 2024-09-17T15:46:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254effd640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7f2550135c80 legacy=0x7f255013df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:46:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254f7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f25500a54b0 0x7f25500a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35772/0 (socket says 172.21.15.64:35772) 2024-09-17T15:46:26.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254f7fe640 1 -- 172.21.15.64:0/2032284347 learned_addr learned my addr 172.21.15.64:0/2032284347 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:26.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254e7fc640 1 -- 172.21.15.64:0/2032284347 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1690465058 0 0) 0x7f255009da60 con 0x7f25501360a0 2024-09-17T15:46:26.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254e7fc640 10 monclient(hunting): my global_id is 4544 2024-09-17T15:46:26.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254e7fc640 1 -- 172.21.15.64:0/2032284347 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2538003520 con 0x7f25501360a0 2024-09-17T15:46:26.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254e7fc640 1 -- 172.21.15.64:0/2032284347 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2703034622 0 0) 0x7f25500aacb0 con 0x7f25500a54b0 2024-09-17T15:46:26.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254e7fc640 10 monclient(hunting): my global_id is 4529 2024-09-17T15:46:26.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.452+0000 7f254e7fc640 1 -- 172.21.15.64:0/2032284347 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f255009da60 con 0x7f25500a54b0 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 1 -- 172.21.15.64:0/2032284347 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3799500329 0 0) 0x7f255009da60 con 0x7f25500a54b0 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 1 -- 172.21.15.64:0/2032284347 >> v1:172.21.15.64:6793/0 conn(0x7f25501360a0 legacy=0x7f2550141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 1 -- 172.21.15.64:0/2032284347 >> v1:172.21.15.64:6792/0 conn(0x7f2550135c80 legacy=0x7f255013df10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 1 monclient: found mon.noname-b 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 1 -- 172.21.15.64:0/2032284347 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255009dd40 con 0x7f25500a54b0 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 1 -- 172.21.15.64:0/2032284347 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2540001810 con 0x7f25500a54b0 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 10 monclient: dump: 2024-09-17T15:46:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.469 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f2555c39640 5 monclient: authenticate success, global_id 4529 2024-09-17T15:46:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f2555c39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 1 -- 172.21.15.64:0/2032284347 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2540001a30 con 0x7f25500a54b0 2024-09-17T15:46:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 1 -- 172.21.15.64:0/2032284347 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2540004be0 con 0x7f25500a54b0 2024-09-17T15:46:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f254e7fc640 10 monclient: dump: 2024-09-17T15:46:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.470 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.471 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f2555c39640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f2555c39640 4 set_mon_vals no callback set 2024-09-17T15:46:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f2555c39640 10 monclient: shutdown 2024-09-17T15:46:26.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f2555c39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:26.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f2555c39640 1 -- 172.21.15.64:0/2032284347 >> v1:172.21.15.64:6790/0 conn(0x7f25500a54b0 legacy=0x7f25500a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f2555c39640 1 -- 172.21.15.64:0/2032284347 shutdown_connections 2024-09-17T15:46:26.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f2555c39640 1 -- 172.21.15.64:0/2032284347 >> 172.21.15.64:0/2032284347 conn(0x7f255006daa0 msgr2=0x7f25500ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:26.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f2555c39640 1 -- 172.21.15.64:0/2032284347 shutdown_connections 2024-09-17T15:46:26.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.453+0000 7f2555c39640 1 -- 172.21.15.64:0/2032284347 wait complete. 2024-09-17T15:46:26.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.454+0000 7f2555c39640 10 monclient: build_initial_monmap 2024-09-17T15:46:26.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.454+0000 7f2555c39640 10 monclient: monmap: 2024-09-17T15:46:26.472 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:26.472 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:26.472 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:26.472 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:26.473 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.454+0000 7f2555c39640 1 Processor -- start 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2555c39640 1 -- start start 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2555c39640 10 monclient: init 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2555c39640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2555c39640 10 monclient: _add_conns ranks=[1,5,2,6,7,0,3,4,8] 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2555c39640 10 monclient(hunting): picked mon.noname-f con 0x7f25500a54b0 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2555c39640 10 monclient(hunting): picked mon.noname-h con 0x7f2550135c80 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2555c39640 10 monclient(hunting): picked mon.noname-b con 0x7f25501360a0 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2555c39640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25500a0de0 con 0x7f25500a54b0 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2555c39640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f255009f120 con 0x7f25501360a0 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2555c39640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25500a66b0 con 0x7f2550135c80 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2555c39640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2555c39640 10 monclient(hunting): authenticate will time out at 784.131554s 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f254effd640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7f2550135c80 legacy=0x7f255009ea10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f254ffff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f25501360a0 0x7f25501d6240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35784/0 (socket says 172.21.15.64:35784) 2024-09-17T15:46:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f254ffff640 1 -- 172.21.15.64:0/3162766419 learned_addr learned my addr 172.21.15.64:0/3162766419 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3961040967 0 0) 0x7f255009f120 con 0x7f25501360a0 2024-09-17T15:46:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2537fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2537fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2537fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2537fff640 10 monclient(hunting): my global_id is 4538 2024-09-17T15:46:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.455+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f253c003520 con 0x7f25501360a0 2024-09-17T15:46:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1131786318 0 0) 0x7f253c003520 con 0x7f25501360a0 2024-09-17T15:46:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f255009f120 con 0x7f25501360a0 2024-09-17T15:46:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2548001dd0 con 0x7f25501360a0 2024-09-17T15:46:26.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:26.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:26.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient(hunting): dump: 2024-09-17T15:46:26.476 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.476 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.476 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.476 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.476 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.477 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.477 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.477 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.477 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.477 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.477 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.478 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.478 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.478 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.478 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.478 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:26.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3197001385 0 0) 0x7f25500a0de0 con 0x7f25500a54b0 2024-09-17T15:46:26.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient(hunting): my global_id is 4618 2024-09-17T15:46:26.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f253c003520 con 0x7f25500a54b0 2024-09-17T15:46:26.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 816670971 0 0) 0x7f255009f120 con 0x7f25501360a0 2024-09-17T15:46:26.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 >> v1:172.21.15.100:6791/0 conn(0x7f2550135c80 legacy=0x7f255009ea10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:26.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 >> v1:172.21.15.100:6789/0 conn(0x7f25500a54b0 legacy=0x7f255009e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:26.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 1 monclient: found mon.b 2024-09-17T15:46:26.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:26.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:26.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25501328f0 con 0x7f25501360a0 2024-09-17T15:46:26.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient: _finish_auth 0 2024-09-17T15:46:26.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:26.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:26.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2555c39640 5 monclient: authenticate success, global_id 4538 2024-09-17T15:46:26.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2555c39640 10 monclient: _renew_subs 2024-09-17T15:46:26.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2555c39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:26.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2555c39640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f25500a0b00 con 0x7f25501360a0 2024-09-17T15:46:26.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2555c39640 10 monclient: _renew_subs 2024-09-17T15:46:26.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2555c39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:46:26.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2555c39640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f255009f5e0 con 0x7f25501360a0 2024-09-17T15:46:26.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2548001ff0 con 0x7f25501360a0 2024-09-17T15:46:26.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:26.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2548003f60 con 0x7f25501360a0 2024-09-17T15:46:26.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f254dffb640 4 set_mon_vals no callback set 2024-09-17T15:46:26.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:26.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.456+0000 7f2537fff640 10 monclient: dump: 2024-09-17T15:46:26.484 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.484 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.484 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.484 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.484 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.485 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.485 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.485 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.485 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.485 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.485 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.486 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.486 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.486 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.486 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.486 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2555c39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:26.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2555c39640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f25300065c0 con 0x7f2530002860 2024-09-17T15:46:26.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f2548004100 con 0x7f25501360a0 2024-09-17T15:46:26.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 3506+0+0 (unknown 3547758874 0 0) 0x7f254804ec90 con 0x7f25501360a0 2024-09-17T15:46:26.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2991696885 0 0) 0x7f25300065c0 con 0x7f2530002860 2024-09-17T15:46:26.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 10 monclient(hunting): my global_id is 4627 2024-09-17T15:46:26.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f255009f120 con 0x7f2530002860 2024-09-17T15:46:26.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f254effd640 10 monclient: get_auth_request con 0x7f253c040300 auth_method 0 2024-09-17T15:46:26.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 813851228 0 0) 0x7f255009f120 con 0x7f2530002860 2024-09-17T15:46:26.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25300065c0 con 0x7f2530002860 2024-09-17T15:46:26.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f25400017d0 con 0x7f2530002860 2024-09-17T15:46:26.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:26.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.457+0000 7f2537fff640 10 monclient: dump: 2024-09-17T15:46:26.488 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.489 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.489 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.489 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.489 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.489 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.489 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.489 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.490 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.490 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.490 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.490 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.490 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.490 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.491 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.491 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.458+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1481434550 0 0) 0x7f25300065c0 con 0x7f2530002860 2024-09-17T15:46:26.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.458+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.100:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f253000a160 con 0x7f2530002860 2024-09-17T15:46:26.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.458+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f25400049d0 con 0x7f2530002860 2024-09-17T15:46:26.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.458+0000 7f2537fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:26.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.458+0000 7f2537fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:26.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.458+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 >> v1:172.21.15.100:6789/0 conn(0x7f2530002860 legacy=0x7f2530004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.458+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 >> v1:172.21.15.100:6789/0 conn(0x7f2530002860 legacy=0x7f2530004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:26.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2555c39640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:26.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2555c39640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2530009e70 con 0x7f2530009130 2024-09-17T15:46:26.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 138723721 0 0) 0x7f2530009e70 con 0x7f2530009130 2024-09-17T15:46:26.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 10 monclient(hunting): my global_id is 4636 2024-09-17T15:46:26.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25300065c0 con 0x7f2530009130 2024-09-17T15:46:26.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1488233661 0 0) 0x7f25300065c0 con 0x7f2530009130 2024-09-17T15:46:26.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2530009e70 con 0x7f2530009130 2024-09-17T15:46:26.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2540005000 con 0x7f2530009130 2024-09-17T15:46:26.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:26.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 10 monclient: dump: 2024-09-17T15:46:26.496 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.496 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.496 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.497 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.497 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.497 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.497 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.497 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.497 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.498 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.498 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.498 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.498 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.498 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.498 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.498 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1856303312 0 0) 0x7f2530009e70 con 0x7f2530009130 2024-09-17T15:46:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 --> v1:172.21.15.100:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f253000c410 con 0x7f2530009130 2024-09-17T15:46:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7677 (unknown 0 0 3265062329) 0x7f2540006470 con 0x7f2530009130 2024-09-17T15:46:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 >> v1:172.21.15.100:6789/0 conn(0x7f2530009130 legacy=0x7f25300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.465+0000 7f2537fff640 1 -- 172.21.15.64:0/3162766419 >> v1:172.21.15.100:6789/0 conn(0x7f2530009130 legacy=0x7f25300095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:26.499 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:46:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 12, 2024-09-17T15:46:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 43117, 2024-09-17T15:46:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:46:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:46:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:46:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:46:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:26.521 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:26.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.466+0000 7f2555c39640 1 -- 172.21.15.64:0/3162766419 >> v1:172.21.15.100:6812/4210963484 conn(0x7f253c040300 legacy=0x7f253c0427c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.466+0000 7f2555c39640 10 monclient: shutdown 2024-09-17T15:46:26.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.466+0000 7f2555c39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.466+0000 7f2555c39640 1 -- 172.21.15.64:0/3162766419 >> v1:172.21.15.64:6790/0 conn(0x7f25501360a0 legacy=0x7f25501d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.466+0000 7f254ffff640 1 -- 172.21.15.64:0/3162766419 reap_dead start 2024-09-17T15:46:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.466+0000 7f254ffff640 1 -- 172.21.15.64:0/3162766419 reap_dead start 2024-09-17T15:46:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.466+0000 7f2555c39640 1 -- 172.21.15.64:0/3162766419 shutdown_connections 2024-09-17T15:46:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.466+0000 7f2555c39640 1 -- 172.21.15.64:0/3162766419 >> 172.21.15.64:0/3162766419 conn(0x7f255006daa0 msgr2=0x7f25500a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.466+0000 7f2555c39640 1 -- 172.21.15.64:0/3162766419 shutdown_connections 2024-09-17T15:46:26.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.466+0000 7f2555c39640 1 -- 172.21.15.64:0/3162766419 wait complete. 2024-09-17T15:46:26.524 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-09-17T15:46:26.524 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon i 2024-09-17T15:46:26.525 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-17T15:46:26.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.626+0000 7ff7221c2640 10 monclient: get_monmap_and_config 2024-09-17T15:46:26.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.626+0000 7ff7221c2640 10 monclient: build_initial_monmap 2024-09-17T15:46:26.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.626+0000 7ff7221c2640 10 monclient: monmap: 2024-09-17T15:46:26.628 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:26.628 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.628 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:46:26.627610+0000 2024-09-17T15:46:26.628 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:46:26.627610+0000 2024-09-17T15:46:26.628 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:26.629 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.627+0000 7ff7221c2640 1 Processor -- start 2024-09-17T15:46:26.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.627+0000 7ff7221c2640 1 -- start start 2024-09-17T15:46:26.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.627+0000 7ff7221c2640 10 monclient: init 2024-09-17T15:46:26.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.627+0000 7ff7221c2640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:26.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.627+0000 7ff7221c2640 10 monclient: _add_conns ranks=[0,2,1,6,4,7,3,5,8] 2024-09-17T15:46:26.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff7221c2640 10 monclient(hunting): picked mon.noname-a con 0x7ff71c0a54b0 addr v1:172.21.15.64:6789/0 2024-09-17T15:46:26.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff7221c2640 10 monclient(hunting): picked mon.noname-b con 0x7ff71c135c80 addr v1:172.21.15.64:6790/0 2024-09-17T15:46:26.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff7221c2640 10 monclient(hunting): picked mon.noname-f con 0x7ff71c1360a0 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:26.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff7221c2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff71c0aacb0 con 0x7ff71c0a54b0 2024-09-17T15:46:26.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff7221c2640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff71c09d780 con 0x7ff71c1360a0 2024-09-17T15:46:26.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff7221c2640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff71c09da60 con 0x7ff71c135c80 2024-09-17T15:46:26.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff7221c2640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:26.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff7221c2640 10 monclient(hunting): authenticate will time out at 784.303087s 2024-09-17T15:46:26.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71b7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7ff71c0a54b0 0x7ff71c0a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:44046/0 (socket says 172.21.15.64:44046) 2024-09-17T15:46:26.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71b7fe640 1 -- 172.21.15.64:0/338111323 learned_addr learned my addr 172.21.15.64:0/338111323 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:26.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 1 -- 172.21.15.64:0/338111323 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1768497016 0 0) 0x7ff71c0aacb0 con 0x7ff71c0a54b0 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient(hunting): my global_id is 4518 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 1 -- 172.21.15.64:0/338111323 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff704003520 con 0x7ff71c0a54b0 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 1 -- 172.21.15.64:0/338111323 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 234707161 0 0) 0x7ff71c09da60 con 0x7ff71c135c80 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient(hunting): my global_id is 4547 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 1 -- 172.21.15.64:0/338111323 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff71c0aacb0 con 0x7ff71c135c80 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 1 -- 172.21.15.64:0/338111323 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2757369657 0 0) 0x7ff71c0aacb0 con 0x7ff71c135c80 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 1 -- 172.21.15.64:0/338111323 >> v1:172.21.15.100:6789/0 conn(0x7ff71c1360a0 legacy=0x7ff71c141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 1 -- 172.21.15.64:0/338111323 >> v1:172.21.15.64:6789/0 conn(0x7ff71c0a54b0 legacy=0x7ff71c0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 1 monclient: found mon.noname-b 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 1 -- 172.21.15.64:0/338111323 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71c09dd40 con 0x7ff71c135c80 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 1 -- 172.21.15.64:0/338111323 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff710001790 con 0x7ff71c135c80 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient: dump: 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.639 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.640 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff7221c2640 5 monclient: authenticate success, global_id 4547 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff7221c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 1 -- 172.21.15.64:0/338111323 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff710001930 con 0x7ff71c135c80 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 1 -- 172.21.15.64:0/338111323 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff710003610 con 0x7ff71c135c80 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff7221c2640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff7221c2640 4 set_mon_vals no callback set 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.628+0000 7ff71a7fc640 10 monclient: dump: 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.641 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.629+0000 7ff7221c2640 10 monclient: shutdown 2024-09-17T15:46:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.629+0000 7ff7221c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.629+0000 7ff7221c2640 1 -- 172.21.15.64:0/338111323 >> v1:172.21.15.64:6790/0 conn(0x7ff71c135c80 legacy=0x7ff71c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.629+0000 7ff7221c2640 1 -- 172.21.15.64:0/338111323 shutdown_connections 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.629+0000 7ff7221c2640 1 -- 172.21.15.64:0/338111323 >> 172.21.15.64:0/338111323 conn(0x7ff71c06daa0 msgr2=0x7ff71c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.629+0000 7ff7221c2640 1 -- 172.21.15.64:0/338111323 shutdown_connections 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.629+0000 7ff7221c2640 1 -- 172.21.15.64:0/338111323 wait complete. 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.629+0000 7ff7221c2640 10 monclient: build_initial_monmap 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.629+0000 7ff7221c2640 10 monclient: monmap: 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:46:26.643 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff7221c2640 1 Processor -- start 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff7221c2640 1 -- start start 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff7221c2640 10 monclient: init 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff7221c2640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:26.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff7221c2640 10 monclient: _add_conns ranks=[6,1,5,2,4,8,7,3,0] 2024-09-17T15:46:26.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff7221c2640 10 monclient(hunting): picked mon.noname-d con 0x7ff71c0a54b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:46:26.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff7221c2640 10 monclient(hunting): picked mon.noname-f con 0x7ff71c135c80 addr v1:172.21.15.100:6789/0 2024-09-17T15:46:26.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff7221c2640 10 monclient(hunting): picked mon.noname-h con 0x7ff71c1360a0 addr v1:172.21.15.100:6791/0 2024-09-17T15:46:26.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff7221c2640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff71c0a2440 con 0x7ff71c135c80 2024-09-17T15:46:26.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff71b7fe640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7ff71c0a54b0 legacy=0x7ff71c09f060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:46:26.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff7221c2640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff71c0a2720 con 0x7ff71c1360a0 2024-09-17T15:46:26.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff7221c2640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff71c0a06a0 con 0x7ff71c0a54b0 2024-09-17T15:46:26.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff7221c2640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:26.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.630+0000 7ff7221c2640 10 monclient(hunting): authenticate will time out at 784.305982s 2024-09-17T15:46:26.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff71bfff640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7ff71c1360a0 legacy=0x7ff71c1d5e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:46:26.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff71affd640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff71c135c80 0x7ff71c09f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.64:32970/0 (socket says 172.21.15.64:32970) 2024-09-17T15:46:26.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff71affd640 1 -- 172.21.15.64:0/1330520777 learned_addr learned my addr 172.21.15.64:0/1330520777 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:46:26.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1670736264 0 0) 0x7ff71c0a2440 con 0x7ff71c135c80 2024-09-17T15:46:26.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 10 monclient(hunting): my global_id is 4654 2024-09-17T15:46:26.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff708003520 con 0x7ff71c135c80 2024-09-17T15:46:26.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1130355719 0 0) 0x7ff708003520 con 0x7ff71c135c80 2024-09-17T15:46:26.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff71c0a2440 con 0x7ff71c135c80 2024-09-17T15:46:26.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff710001650 con 0x7ff71c135c80 2024-09-17T15:46:26.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:46:26.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:46:26.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 10 monclient(hunting): dump: 2024-09-17T15:46:26.648 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.649 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.649 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.649 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.649 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.649 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.649 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.649 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.649 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.649 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.649 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.650 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.650 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.650 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.650 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.650 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:26.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.631+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4128427298 0 0) 0x7ff71c0a2440 con 0x7ff71c135c80 2024-09-17T15:46:26.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 >> v1:172.21.15.64:6792/0 conn(0x7ff71c0a54b0 legacy=0x7ff71c09f060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:26.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 >> v1:172.21.15.100:6791/0 conn(0x7ff71c1360a0 legacy=0x7ff71c1d5e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:26.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:26.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 1 monclient: found mon.f 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71c0a2a00 con 0x7ff71c135c80 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 10 monclient: _finish_auth 0 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff7221c2640 5 monclient: authenticate success, global_id 4654 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff7221c2640 10 monclient: _renew_subs 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff7221c2640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff7221c2640 1 -- 172.21.15.64:0/1330520777 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff71c0a5190 con 0x7ff71c135c80 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff7221c2640 10 monclient: _renew_subs 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff7221c2640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.100:6789/0 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff7221c2640 1 -- 172.21.15.64:0/1330520777 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff71c0a0060 con 0x7ff71c135c80 2024-09-17T15:46:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.1 v1:172.21.15.100:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff710001870 con 0x7ff71c135c80 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.1 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff710003e40 con 0x7ff71c135c80 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff719ffb640 4 set_mon_vals no callback set 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff703fff640 10 monclient: dump: 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.653 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff7221c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.632+0000 7ff7221c2640 1 -- 172.21.15.64:0/1330520777 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff6fc0065c0 con 0x7ff6fc002860 2024-09-17T15:46:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.1 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7ff710004060 con 0x7ff71c135c80 2024-09-17T15:46:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.1 v1:172.21.15.100:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 3506+0+0 (unknown 3547758874 0 0) 0x7ff71004ec20 con 0x7ff71c135c80 2024-09-17T15:46:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3537555420 0 0) 0x7ff6fc0065c0 con 0x7ff6fc002860 2024-09-17T15:46:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 10 monclient(hunting): my global_id is 4507 2024-09-17T15:46:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff71c0a2440 con 0x7ff6fc002860 2024-09-17T15:46:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff71bfff640 10 monclient: get_auth_request con 0x7ff7080400f0 auth_method 0 2024-09-17T15:46:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 930864686 0 0) 0x7ff71c0a2440 con 0x7ff6fc002860 2024-09-17T15:46:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6fc0065c0 con 0x7ff6fc002860 2024-09-17T15:46:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff70c0016a0 con 0x7ff6fc002860 2024-09-17T15:46:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:46:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 10 monclient: dump: 2024-09-17T15:46:26.654 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.655 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2616663207 0 0) 0x7ff6fc0065c0 con 0x7ff6fc002860 2024-09-17T15:46:26.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.633+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 --> v1:172.21.15.100:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6fc00a160 con 0x7ff6fc002860 2024-09-17T15:46:26.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.634+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff70c001cf0 con 0x7ff6fc002860 2024-09-17T15:46:26.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.634+0000 7ff703fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:26.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.634+0000 7ff703fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:26.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.634+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 >> v1:172.21.15.100:6792/0 conn(0x7ff6fc002860 legacy=0x7ff6fc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.634+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 >> v1:172.21.15.100:6792/0 conn(0x7ff6fc002860 legacy=0x7ff6fc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:26.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.640+0000 7ff7221c2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:26.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.640+0000 7ff7221c2640 1 -- 172.21.15.64:0/1330520777 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff6fc009e50 con 0x7ff6fc009110 2024-09-17T15:46:26.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.640+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2031375708 0 0) 0x7ff6fc009e50 con 0x7ff6fc009110 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.640+0000 7ff703fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.640+0000 7ff703fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.640+0000 7ff703fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 10 monclient(hunting): my global_id is 4516 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6fc0065c0 con 0x7ff6fc009110 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3645332791 0 0) 0x7ff6fc0065c0 con 0x7ff6fc009110 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6fc009e50 con 0x7ff6fc009110 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff70c004c60 con 0x7ff6fc009110 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 10 monclient: dump: 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:46:26.658 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 973636173 0 0) 0x7ff6fc009e50 con 0x7ff6fc009110 2024-09-17T15:46:26.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 --> v1:172.21.15.100:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff6fc00c3f0 con 0x7ff6fc009110 2024-09-17T15:46:26.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7677 (unknown 0 0 1017038755) 0x7ff70c006150 con 0x7ff6fc009110 2024-09-17T15:46:26.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:26.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:26.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 >> v1:172.21.15.100:6792/0 conn(0x7ff6fc009110 legacy=0x7ff6fc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.641+0000 7ff703fff640 1 -- 172.21.15.64:0/1330520777 >> v1:172.21.15.100:6792/0 conn(0x7ff6fc009110 legacy=0x7ff6fc0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:26.660 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:46:26.660 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:26.660 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:26.660 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 12, 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 43276, 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:26.661 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:26.662 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.663 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:46:26.664 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:46:26.664 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:46:26.664 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:46:26.664 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:46:26.664 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:46:26.664 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:26.664 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:46:26.664 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:46:26.664 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:26.664 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:46:26.664 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:26.664 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:46:26.664 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:46:26.665 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.668 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.668 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:46:26.668 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.668 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.668 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:46:26.668 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:46:26.668 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.668 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.669 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.669 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:46:26.669 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:46:26.669 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.669 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.669 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.670 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:46:26.670 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.670 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.670 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:46:26.670 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:46:26.670 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.670 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.671 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.671 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:46:26.671 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:46:26.671 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.671 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.671 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.672 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:46:26.672 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.672 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.672 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:46:26.672 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:46:26.672 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.673 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.673 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.673 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:46:26.673 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:46:26.673 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.673 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.674 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.674 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:46:26.674 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.674 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.674 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.674 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:46:26.674 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:46:26.674 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.675 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.675 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.675 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:46:26.675 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:46:26.675 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.675 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.676 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.676 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:46:26.676 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.676 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.676 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.676 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:46:26.676 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:46:26.677 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.677 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.677 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.677 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:46:26.677 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:46:26.677 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.678 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.678 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.678 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:46:26.678 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.678 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.678 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:46:26.679 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:46:26.679 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.679 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.679 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.679 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:46:26.679 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:46:26.679 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.680 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.680 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.680 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:46:26.680 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.680 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.680 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.680 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:46:26.681 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:46:26.681 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.681 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.681 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.681 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:46:26.681 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:46:26.681 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:46:26.682 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:46:26.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:46:26.682 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:46:26.682 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:46:26.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.682 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.682 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:46:26.683 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:46:26.683 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:46:26.683 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:46:26.683 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:46:26.683 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.683 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.683 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:46:26.683 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:46:26.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.683 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:26.683 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:26.684 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:26.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.684 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.684 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:46:26.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.684 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:26.684 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:26.684 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 4 2024-09-17T15:46:26.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.684 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:46:26.685 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": [ 2024-09-17T15:46:26.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:46:26.685 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:26.685 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:46:26.685 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:46:26.685 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:46:26.685 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:46:26.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:46:26.685 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:46:26.685 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:46:26.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.642+0000 7ff7221c2640 1 -- 172.21.15.64:0/1330520777 >> v1:172.21.15.100:6812/4210963484 conn(0x7ff7080400f0 legacy=0x7ff7080425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.642+0000 7ff7221c2640 10 monclient: shutdown 2024-09-17T15:46:26.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.642+0000 7ff7221c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:26.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.642+0000 7ff7221c2640 1 -- 172.21.15.64:0/1330520777 >> v1:172.21.15.100:6789/0 conn(0x7ff71c135c80 legacy=0x7ff71c09f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:26.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.642+0000 7ff71bfff640 1 -- 172.21.15.64:0/1330520777 reap_dead start 2024-09-17T15:46:26.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.642+0000 7ff71bfff640 1 -- 172.21.15.64:0/1330520777 reap_dead start 2024-09-17T15:46:26.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.642+0000 7ff7221c2640 1 -- 172.21.15.64:0/1330520777 shutdown_connections 2024-09-17T15:46:26.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.642+0000 7ff7221c2640 1 -- 172.21.15.64:0/1330520777 >> 172.21.15.64:0/1330520777 conn(0x7ff71c06daa0 msgr2=0x7ff71c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:26.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.642+0000 7ff7221c2640 1 -- 172.21.15.64:0/1330520777 shutdown_connections 2024-09-17T15:46:26.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:46:26.642+0000 7ff7221c2640 1 -- 172.21.15.64:0/1330520777 wait complete. 2024-09-17T15:46:26.688 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-09-17T15:46:26.688 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-17T15:46:28.495 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:28.495 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35005]: starting. 2024-09-17T15:46:28.495 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35005]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-34970 2024-09-17T15:46:28.495 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35005]: created object 0... 2024-09-17T15:46:28.495 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35005]: created object 25... 2024-09-17T15:46:28.496 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35005]: created object 49... 2024-09-17T15:46:28.496 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35005]: finishing. 2024-09-17T15:46:28.496 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35005]: shutting down. 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35007]: starting. 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35007]: listing objects. 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35007]: listed object 0... 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35007]: listed object 25... 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35007]: saw 50 objects 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35007]: shutting down. 2024-09-17T15:46:30.431 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:30.431 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35006]: starting. 2024-09-17T15:46:30.431 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35006]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-34970 2024-09-17T15:46:30.431 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35006]: shutting down. 2024-09-17T15:46:30.432 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:30.433 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:30.433 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:46:30.435 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:46:32.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35074]: starting. 2024-09-17T15:46:32.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35074]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35073 2024-09-17T15:46:32.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35074]: created object 0... 2024-09-17T15:46:32.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35074]: created object 25... 2024-09-17T15:46:32.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35074]: created object 49... 2024-09-17T15:46:32.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35074]: finishing. 2024-09-17T15:46:32.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35074]: shutting down. 2024-09-17T15:46:33.443 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35075]: starting. 2024-09-17T15:46:33.443 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35075]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35073 2024-09-17T15:46:33.443 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35075]: shutting down. 2024-09-17T15:46:35.553 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:35.554 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35108]: starting. 2024-09-17T15:46:35.554 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35108]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35073 2024-09-17T15:46:35.554 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35108]: created object 0... 2024-09-17T15:46:35.554 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35108]: created object 25... 2024-09-17T15:46:35.554 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35108]: created object 49... 2024-09-17T15:46:35.554 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35108]: finishing. 2024-09-17T15:46:35.554 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35108]: shutting down. 2024-09-17T15:46:35.564 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:35.564 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35110]: starting. 2024-09-17T15:46:35.564 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35110]: listing objects. 2024-09-17T15:46:35.564 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35110]: listed object 0... 2024-09-17T15:46:35.564 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35110]: listed object 25... 2024-09-17T15:46:35.564 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35110]: saw 50 objects 2024-09-17T15:46:35.565 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35110]: shutting down. 2024-09-17T15:46:36.465 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:36.466 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35109]: starting. 2024-09-17T15:46:36.466 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35109]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35073 2024-09-17T15:46:36.466 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35109]: shutting down. 2024-09-17T15:46:36.467 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:36.467 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:36.467 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:46:36.474 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:46:41.477 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35177]: starting. 2024-09-17T15:46:41.477 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35177]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35176 2024-09-17T15:46:41.477 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35177]: created object 0... 2024-09-17T15:46:41.477 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35177]: created object 25... 2024-09-17T15:46:41.477 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35177]: created object 49... 2024-09-17T15:46:41.477 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35177]: finishing. 2024-09-17T15:46:41.478 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35177]: shutting down. 2024-09-17T15:46:42.388 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35178]: starting. 2024-09-17T15:46:42.388 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35178]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35176 2024-09-17T15:46:42.389 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35178]: shutting down. 2024-09-17T15:46:44.506 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:44.506 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35211]: starting. 2024-09-17T15:46:44.506 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35211]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35176 2024-09-17T15:46:44.506 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35211]: created object 0... 2024-09-17T15:46:44.506 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35211]: created object 25... 2024-09-17T15:46:44.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35211]: created object 49... 2024-09-17T15:46:44.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35211]: finishing. 2024-09-17T15:46:44.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35211]: shutting down. 2024-09-17T15:46:44.517 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:44.518 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35213]: starting. 2024-09-17T15:46:44.518 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35213]: listing objects. 2024-09-17T15:46:44.518 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35213]: listed object 0... 2024-09-17T15:46:44.518 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35213]: listed object 25... 2024-09-17T15:46:44.518 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35213]: saw 50 objects 2024-09-17T15:46:44.518 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35213]: shutting down. 2024-09-17T15:46:45.409 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:45.409 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35212]: starting. 2024-09-17T15:46:45.409 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35212]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35176 2024-09-17T15:46:45.409 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35212]: shutting down. 2024-09-17T15:46:45.410 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:45.411 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:45.411 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:46:45.414 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:46:47.523 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35280]: starting. 2024-09-17T15:46:47.523 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35280]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35279 2024-09-17T15:46:47.523 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35280]: created object 0... 2024-09-17T15:46:47.523 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35280]: created object 25... 2024-09-17T15:46:47.523 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35280]: created object 49... 2024-09-17T15:46:47.524 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35280]: finishing. 2024-09-17T15:46:47.524 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35280]: shutting down. 2024-09-17T15:46:48.368 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35281]: starting. 2024-09-17T15:46:48.369 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35281]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35279 2024-09-17T15:46:48.369 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35281]: shutting down. 2024-09-17T15:46:50.488 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:50.488 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35314]: starting. 2024-09-17T15:46:50.488 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35314]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35279 2024-09-17T15:46:50.488 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35314]: created object 0... 2024-09-17T15:46:50.488 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35314]: created object 25... 2024-09-17T15:46:50.488 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35314]: created object 49... 2024-09-17T15:46:50.489 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35314]: finishing. 2024-09-17T15:46:50.489 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35314]: shutting down. 2024-09-17T15:46:50.499 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:50.499 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35316]: starting. 2024-09-17T15:46:50.499 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35316]: listing objects. 2024-09-17T15:46:50.499 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35316]: listed object 0... 2024-09-17T15:46:50.499 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35316]: listed object 25... 2024-09-17T15:46:50.499 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35316]: saw 50 objects 2024-09-17T15:46:50.499 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35316]: shutting down. 2024-09-17T15:46:51.391 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:51.391 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35315]: starting. 2024-09-17T15:46:51.391 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35315]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35279 2024-09-17T15:46:51.391 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35315]: shutting down. 2024-09-17T15:46:51.392 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:51.393 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:51.393 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:46:51.395 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:46:53.460 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35383]: starting. 2024-09-17T15:46:53.460 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35383]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35382 2024-09-17T15:46:53.460 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35383]: created object 0... 2024-09-17T15:46:53.460 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35383]: created object 25... 2024-09-17T15:46:53.461 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35383]: created object 49... 2024-09-17T15:46:53.461 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35383]: finishing. 2024-09-17T15:46:53.461 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35383]: shutting down. 2024-09-17T15:46:54.373 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35384]: starting. 2024-09-17T15:46:54.374 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35384]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35382 2024-09-17T15:46:54.374 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35384]: shutting down. 2024-09-17T15:46:56.496 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:56.496 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35417]: starting. 2024-09-17T15:46:56.496 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35417]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35382 2024-09-17T15:46:56.496 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35417]: created object 0... 2024-09-17T15:46:56.496 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35417]: created object 25... 2024-09-17T15:46:56.496 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35417]: created object 49... 2024-09-17T15:46:56.497 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35417]: finishing. 2024-09-17T15:46:56.497 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35417]: shutting down. 2024-09-17T15:46:56.507 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:56.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35419]: starting. 2024-09-17T15:46:56.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35419]: listing objects. 2024-09-17T15:46:56.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35419]: listed object 0... 2024-09-17T15:46:56.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35419]: listed object 25... 2024-09-17T15:46:56.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35419]: saw 50 objects 2024-09-17T15:46:56.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35419]: shutting down. 2024-09-17T15:46:57.414 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:57.414 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35418]: starting. 2024-09-17T15:46:57.414 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35418]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35382 2024-09-17T15:46:57.414 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35418]: shutting down. 2024-09-17T15:46:57.415 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:57.416 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:46:57.416 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:46:57.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35486]: starting. 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35486]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35485 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35486]: created object 0... 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35486]: created object 25... 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35486]: created object 49... 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35486]: finishing. 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35486]: shutting down. 2024-09-17T15:47:00.445 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35487]: starting. 2024-09-17T15:47:00.446 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35487]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35485 2024-09-17T15:47:00.446 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35487]: shutting down. 2024-09-17T15:47:02.566 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:02.566 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35520]: starting. 2024-09-17T15:47:02.566 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35520]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35485 2024-09-17T15:47:02.566 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35520]: created object 0... 2024-09-17T15:47:02.566 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35520]: created object 25... 2024-09-17T15:47:02.566 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35520]: created object 49... 2024-09-17T15:47:02.567 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35520]: finishing. 2024-09-17T15:47:02.567 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35520]: shutting down. 2024-09-17T15:47:02.578 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:02.578 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35522]: starting. 2024-09-17T15:47:02.578 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35522]: listing objects. 2024-09-17T15:47:02.578 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35522]: listed object 0... 2024-09-17T15:47:02.578 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35522]: listed object 25... 2024-09-17T15:47:02.578 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35522]: saw 50 objects 2024-09-17T15:47:02.578 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35522]: shutting down. 2024-09-17T15:47:03.372 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:03.372 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35521]: starting. 2024-09-17T15:47:03.372 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35521]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35485 2024-09-17T15:47:03.373 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35521]: shutting down. 2024-09-17T15:47:03.374 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:03.374 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:03.374 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:47:03.377 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:47:05.468 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35589]: starting. 2024-09-17T15:47:05.469 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35589]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35588 2024-09-17T15:47:05.469 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35589]: created object 0... 2024-09-17T15:47:05.469 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35589]: created object 25... 2024-09-17T15:47:05.469 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35589]: created object 49... 2024-09-17T15:47:05.469 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35589]: finishing. 2024-09-17T15:47:05.469 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35589]: shutting down. 2024-09-17T15:47:06.400 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35590]: starting. 2024-09-17T15:47:06.401 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35590]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35588 2024-09-17T15:47:06.401 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35590]: shutting down. 2024-09-17T15:47:08.487 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:08.488 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35623]: starting. 2024-09-17T15:47:08.488 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35623]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35588 2024-09-17T15:47:08.488 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35623]: created object 0... 2024-09-17T15:47:08.488 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35623]: created object 25... 2024-09-17T15:47:08.488 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35623]: created object 49... 2024-09-17T15:47:08.488 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35623]: finishing. 2024-09-17T15:47:08.488 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35623]: shutting down. 2024-09-17T15:47:08.498 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:08.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35625]: starting. 2024-09-17T15:47:08.499 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35625]: listing objects. 2024-09-17T15:47:08.499 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35625]: listed object 0... 2024-09-17T15:47:08.499 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35625]: listed object 25... 2024-09-17T15:47:08.499 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35625]: saw 50 objects 2024-09-17T15:47:08.499 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35625]: shutting down. 2024-09-17T15:47:09.410 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:09.410 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35624]: starting. 2024-09-17T15:47:09.410 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35624]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35588 2024-09-17T15:47:09.410 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35624]: shutting down. 2024-09-17T15:47:09.412 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:09.412 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:09.412 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:47:09.415 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:47:11.503 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35692]: starting. 2024-09-17T15:47:11.503 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35692]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35691 2024-09-17T15:47:11.503 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35692]: created object 0... 2024-09-17T15:47:11.503 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35692]: created object 25... 2024-09-17T15:47:11.503 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35692]: created object 49... 2024-09-17T15:47:11.503 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35692]: finishing. 2024-09-17T15:47:11.503 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35692]: shutting down. 2024-09-17T15:47:12.435 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35693]: starting. 2024-09-17T15:47:12.435 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35693]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35691 2024-09-17T15:47:12.435 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35693]: shutting down. 2024-09-17T15:47:14.461 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:14.461 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35726]: starting. 2024-09-17T15:47:14.461 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35726]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35691 2024-09-17T15:47:14.461 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35726]: created object 0... 2024-09-17T15:47:14.462 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35726]: created object 25... 2024-09-17T15:47:14.462 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35726]: created object 49... 2024-09-17T15:47:14.462 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35726]: finishing. 2024-09-17T15:47:14.462 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35726]: shutting down. 2024-09-17T15:47:15.471 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:15.471 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35728]: starting. 2024-09-17T15:47:15.471 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35728]: listing objects. 2024-09-17T15:47:15.471 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35728]: listed object 0... 2024-09-17T15:47:15.471 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35728]: listed object 25... 2024-09-17T15:47:15.471 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35728]: saw 50 objects 2024-09-17T15:47:15.472 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35728]: shutting down. 2024-09-17T15:47:16.394 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:16.394 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35727]: starting. 2024-09-17T15:47:16.394 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35727]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35691 2024-09-17T15:47:16.395 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35727]: shutting down. 2024-09-17T15:47:16.396 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:16.396 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:16.396 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:47:16.398 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:47:21.501 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35795]: starting. 2024-09-17T15:47:21.501 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35795]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35794 2024-09-17T15:47:21.501 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35795]: created object 0... 2024-09-17T15:47:21.501 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35795]: created object 25... 2024-09-17T15:47:21.501 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35795]: created object 49... 2024-09-17T15:47:21.501 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35795]: finishing. 2024-09-17T15:47:21.501 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35795]: shutting down. 2024-09-17T15:47:22.440 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35796]: starting. 2024-09-17T15:47:22.440 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35796]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35794 2024-09-17T15:47:22.440 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35796]: shutting down. 2024-09-17T15:47:24.467 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:24.468 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35829]: starting. 2024-09-17T15:47:24.468 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35829]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35794 2024-09-17T15:47:24.468 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35829]: created object 0... 2024-09-17T15:47:24.468 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35829]: created object 25... 2024-09-17T15:47:24.468 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35829]: created object 49... 2024-09-17T15:47:24.468 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35829]: finishing. 2024-09-17T15:47:24.468 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35829]: shutting down. 2024-09-17T15:47:24.478 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:24.478 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35831]: starting. 2024-09-17T15:47:24.479 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35831]: listing objects. 2024-09-17T15:47:24.479 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35831]: listed object 0... 2024-09-17T15:47:24.479 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35831]: listed object 25... 2024-09-17T15:47:24.479 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35831]: saw 50 objects 2024-09-17T15:47:24.479 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35831]: shutting down. 2024-09-17T15:47:25.390 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:25.390 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35830]: starting. 2024-09-17T15:47:25.390 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35830]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35794 2024-09-17T15:47:25.390 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35830]: shutting down. 2024-09-17T15:47:25.392 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:25.392 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:25.392 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:47:25.395 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:47:27.485 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35898]: starting. 2024-09-17T15:47:27.485 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35898]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35897 2024-09-17T15:47:27.485 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35898]: created object 0... 2024-09-17T15:47:27.485 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35898]: created object 25... 2024-09-17T15:47:27.485 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35898]: created object 49... 2024-09-17T15:47:27.485 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35898]: finishing. 2024-09-17T15:47:27.485 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[35898]: shutting down. 2024-09-17T15:47:28.412 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35899]: starting. 2024-09-17T15:47:28.412 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35899]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35897 2024-09-17T15:47:28.412 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[35899]: shutting down. 2024-09-17T15:47:30.508 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:30.508 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35932]: starting. 2024-09-17T15:47:30.508 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35932]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-35897 2024-09-17T15:47:30.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35932]: created object 0... 2024-09-17T15:47:30.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35932]: created object 25... 2024-09-17T15:47:30.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35932]: created object 49... 2024-09-17T15:47:30.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35932]: finishing. 2024-09-17T15:47:30.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[35932]: shutting down. 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35934]: starting. 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35934]: listing objects. 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35934]: listed object 0... 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35934]: listed object 25... 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35934]: saw 50 objects 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[35934]: shutting down. 2024-09-17T15:47:31.428 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:31.428 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35933]: starting. 2024-09-17T15:47:31.428 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35933]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-35897 2024-09-17T15:47:31.428 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[35933]: shutting down. 2024-09-17T15:47:31.430 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:31.430 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:31.430 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:47:31.434 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:47:33.474 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36001]: starting. 2024-09-17T15:47:33.474 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36001]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36000 2024-09-17T15:47:33.474 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36001]: created object 0... 2024-09-17T15:47:33.475 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36001]: created object 25... 2024-09-17T15:47:33.475 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36001]: created object 49... 2024-09-17T15:47:33.475 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36001]: finishing. 2024-09-17T15:47:33.475 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36001]: shutting down. 2024-09-17T15:47:34.384 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36002]: starting. 2024-09-17T15:47:34.384 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36002]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36000 2024-09-17T15:47:34.384 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36002]: shutting down. 2024-09-17T15:47:36.484 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:36.484 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36035]: starting. 2024-09-17T15:47:36.485 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36035]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36000 2024-09-17T15:47:36.485 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36035]: created object 0... 2024-09-17T15:47:36.485 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36035]: created object 25... 2024-09-17T15:47:36.485 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36035]: created object 49... 2024-09-17T15:47:36.485 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36035]: finishing. 2024-09-17T15:47:36.485 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36035]: shutting down. 2024-09-17T15:47:36.495 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:36.495 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36037]: starting. 2024-09-17T15:47:36.495 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36037]: listing objects. 2024-09-17T15:47:36.495 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36037]: listed object 0... 2024-09-17T15:47:36.495 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36037]: listed object 25... 2024-09-17T15:47:36.495 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36037]: saw 50 objects 2024-09-17T15:47:36.495 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36037]: shutting down. 2024-09-17T15:47:37.386 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:37.386 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36036]: starting. 2024-09-17T15:47:37.386 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36036]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36000 2024-09-17T15:47:37.387 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36036]: shutting down. 2024-09-17T15:47:37.388 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:37.388 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:37.388 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:47:37.391 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:47:39.505 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36104]: starting. 2024-09-17T15:47:39.505 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36104]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36103 2024-09-17T15:47:39.505 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36104]: created object 0... 2024-09-17T15:47:39.506 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36104]: created object 25... 2024-09-17T15:47:39.506 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36104]: created object 49... 2024-09-17T15:47:39.506 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36104]: finishing. 2024-09-17T15:47:39.506 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36104]: shutting down. 2024-09-17T15:47:40.431 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36105]: starting. 2024-09-17T15:47:40.431 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36105]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36103 2024-09-17T15:47:40.431 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36105]: shutting down. 2024-09-17T15:47:42.529 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:42.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36138]: starting. 2024-09-17T15:47:42.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36138]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36103 2024-09-17T15:47:42.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36138]: created object 0... 2024-09-17T15:47:42.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36138]: created object 25... 2024-09-17T15:47:42.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36138]: created object 49... 2024-09-17T15:47:42.531 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36138]: finishing. 2024-09-17T15:47:42.531 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36138]: shutting down. 2024-09-17T15:47:42.540 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:42.541 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36140]: starting. 2024-09-17T15:47:42.541 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36140]: listing objects. 2024-09-17T15:47:42.541 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36140]: listed object 0... 2024-09-17T15:47:42.541 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36140]: listed object 25... 2024-09-17T15:47:42.541 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36140]: saw 50 objects 2024-09-17T15:47:42.541 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36140]: shutting down. 2024-09-17T15:47:43.382 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:43.382 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36139]: starting. 2024-09-17T15:47:43.382 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36139]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36103 2024-09-17T15:47:43.382 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36139]: shutting down. 2024-09-17T15:47:43.384 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:43.384 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:43.384 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:47:43.387 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:47:45.537 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36207]: starting. 2024-09-17T15:47:45.537 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36207]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36206 2024-09-17T15:47:45.537 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36207]: created object 0... 2024-09-17T15:47:45.537 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36207]: created object 25... 2024-09-17T15:47:45.537 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36207]: created object 49... 2024-09-17T15:47:45.537 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36207]: finishing. 2024-09-17T15:47:45.537 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36207]: shutting down. 2024-09-17T15:47:46.456 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36208]: starting. 2024-09-17T15:47:46.456 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36208]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36206 2024-09-17T15:47:46.456 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36208]: shutting down. 2024-09-17T15:47:48.547 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:48.547 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36241]: starting. 2024-09-17T15:47:48.547 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36241]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36206 2024-09-17T15:47:48.547 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36241]: created object 0... 2024-09-17T15:47:48.547 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36241]: created object 25... 2024-09-17T15:47:48.547 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36241]: created object 49... 2024-09-17T15:47:48.547 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36241]: finishing. 2024-09-17T15:47:48.547 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36241]: shutting down. 2024-09-17T15:47:48.557 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:48.557 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36243]: starting. 2024-09-17T15:47:48.557 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36243]: listing objects. 2024-09-17T15:47:48.557 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36243]: listed object 0... 2024-09-17T15:47:48.557 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36243]: listed object 25... 2024-09-17T15:47:48.558 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36243]: saw 50 objects 2024-09-17T15:47:48.558 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36243]: shutting down. 2024-09-17T15:47:49.476 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:49.476 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36242]: starting. 2024-09-17T15:47:49.476 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36242]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36206 2024-09-17T15:47:49.476 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36242]: shutting down. 2024-09-17T15:47:49.478 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:49.478 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:49.478 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:47:49.480 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:47:51.582 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36310]: starting. 2024-09-17T15:47:51.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36310]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36309 2024-09-17T15:47:51.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36310]: created object 0... 2024-09-17T15:47:51.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36310]: created object 25... 2024-09-17T15:47:51.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36310]: created object 49... 2024-09-17T15:47:51.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36310]: finishing. 2024-09-17T15:47:51.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36310]: shutting down. 2024-09-17T15:47:52.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36311]: starting. 2024-09-17T15:47:52.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36311]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36309 2024-09-17T15:47:52.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36311]: shutting down. 2024-09-17T15:47:54.483 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:54.483 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36344]: starting. 2024-09-17T15:47:54.483 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36344]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36309 2024-09-17T15:47:54.483 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36344]: created object 0... 2024-09-17T15:47:54.483 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36344]: created object 25... 2024-09-17T15:47:54.483 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36344]: created object 49... 2024-09-17T15:47:54.483 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36344]: finishing. 2024-09-17T15:47:54.483 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36344]: shutting down. 2024-09-17T15:47:54.494 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:54.494 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36346]: starting. 2024-09-17T15:47:54.494 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36346]: listing objects. 2024-09-17T15:47:54.494 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36346]: listed object 0... 2024-09-17T15:47:54.494 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36346]: listed object 25... 2024-09-17T15:47:54.494 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36346]: saw 50 objects 2024-09-17T15:47:54.494 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36346]: shutting down. 2024-09-17T15:47:55.395 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:55.395 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36345]: starting. 2024-09-17T15:47:55.396 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36345]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36309 2024-09-17T15:47:55.396 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36345]: shutting down. 2024-09-17T15:47:55.397 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:55.397 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:47:55.397 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:47:55.400 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:47:56.688 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-09-17T15:47:56.688 INFO:tasks.ceph.mon.g:Restarting daemon 2024-09-17T15:47:56.688 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-09-17T15:47:56.691 INFO:tasks.ceph.mon.g:Started 2024-09-17T15:47:56.691 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-09-17T15:47:56.691 INFO:tasks.ceph.mon.d:Restarting daemon 2024-09-17T15:47:56.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-09-17T15:47:56.694 INFO:tasks.ceph.mon.d:Started 2024-09-17T15:47:56.694 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.h 2024-09-17T15:47:56.695 INFO:tasks.ceph.mon.h:Restarting daemon 2024-09-17T15:47:56.695 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-09-17T15:47:56.697 INFO:tasks.ceph.mon.h:Started 2024-09-17T15:47:56.697 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-17T15:47:56.697 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:47:56.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.811+0000 7f4712a08640 10 monclient: get_monmap_and_config 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.811+0000 7f4712a08640 10 monclient: build_initial_monmap 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.811+0000 7f4712a08640 10 monclient: monmap: 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:47:56.812717+0000 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:47:56.812717+0000 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:47:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:47:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:47:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:47:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:47:56.815 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:47:56.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.812+0000 7f4712a08640 1 Processor -- start 2024-09-17T15:47:56.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.812+0000 7f4712a08640 1 -- start start 2024-09-17T15:47:56.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.812+0000 7f4712a08640 10 monclient: init 2024-09-17T15:47:56.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.813+0000 7f4712a08640 10 monclient: _reopen_session rank -1 2024-09-17T15:47:56.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.813+0000 7f4712a08640 10 monclient: _add_conns ranks=[6,7,0,5,2,4,1,3,8] 2024-09-17T15:47:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.813+0000 7f4712a08640 10 monclient(hunting): picked mon.noname-d con 0x7f470c0a54b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:47:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.813+0000 7f4712a08640 10 monclient(hunting): picked mon.noname-i con 0x7f470c135c80 addr v1:172.21.15.100:6792/0 2024-09-17T15:47:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.813+0000 7f4712a08640 10 monclient(hunting): picked mon.noname-a con 0x7f470c1360a0 addr v1:172.21.15.64:6789/0 2024-09-17T15:47:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.813+0000 7f4712a08640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c0aacb0 con 0x7f470c1360a0 2024-09-17T15:47:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.813+0000 7f470bfff640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f470c0a54b0 0x7f470c0a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:39270/0 (socket says 172.21.15.64:39270) 2024-09-17T15:47:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.813+0000 7f470bfff640 1 -- 172.21.15.64:0/3746576162 learned_addr learned my addr 172.21.15.64:0/3746576162 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:47:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.813+0000 7f4712a08640 1 -- 172.21.15.64:0/3746576162 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c09d780 con 0x7f470c0a54b0 2024-09-17T15:47:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.813+0000 7f4712a08640 1 -- 172.21.15.64:0/3746576162 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c09da60 con 0x7f470c135c80 2024-09-17T15:47:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.813+0000 7f4712a08640 10 monclient(hunting): _renew_subs 2024-09-17T15:47:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:56.813+0000 7f4712a08640 10 monclient(hunting): authenticate will time out at 874.488862s 2024-09-17T15:47:59.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 10 monclient(hunting): tick 2024-09-17T15:47:59.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:47:59.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 1 monclient(hunting): continuing hunt 2024-09-17T15:47:59.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:47:59.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 1 -- 172.21.15.64:0/3746576162 >> v1:172.21.15.100:6792/0 conn(0x7f470c135c80 legacy=0x7f470c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:59.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 1 -- 172.21.15.64:0/3746576162 >> v1:172.21.15.64:6792/0 conn(0x7f470c0a54b0 legacy=0x7f470c0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:59.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 1 -- 172.21.15.64:0/3746576162 >> v1:172.21.15.64:6789/0 conn(0x7f470c1360a0 legacy=0x7f470c141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:59.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 10 monclient: _add_conns ranks=[2,1,4,3,8,5] 2024-09-17T15:47:59.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f46f80037e0 addr v1:172.21.15.64:6790/0 2024-09-17T15:47:59.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 10 monclient(hunting): picked mon.noname-f con 0x7f46f8007470 addr v1:172.21.15.100:6789/0 2024-09-17T15:47:59.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f46f800b1e0 addr v1:172.21.15.64:6791/0 2024-09-17T15:47:59.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 1 -- 172.21.15.64:0/3746576162 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c09dd40 con 0x7f46f8007470 2024-09-17T15:47:59.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 1 -- 172.21.15.64:0/3746576162 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46f800ffe0 con 0x7f46f80037e0 2024-09-17T15:47:59.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 1 -- 172.21.15.64:0/3746576162 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46f8011280 con 0x7f46f800b1e0 2024-09-17T15:47:59.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:47:59.812+0000 7f4709ffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 1 -- 172.21.15.64:0/3746576162 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3243017121 0 0) 0x7f470c09dd40 con 0x7f46f8007470 2024-09-17T15:48:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient(hunting): my global_id is 5104 2024-09-17T15:48:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 1 -- 172.21.15.64:0/3746576162 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46f4003520 con 0x7f46f8007470 2024-09-17T15:48:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 1 -- 172.21.15.64:0/3746576162 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 252360950 0 0) 0x7f46f800ffe0 con 0x7f46f80037e0 2024-09-17T15:48:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient(hunting): my global_id is 4826 2024-09-17T15:48:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 1 -- 172.21.15.64:0/3746576162 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f470c09dd40 con 0x7f46f80037e0 2024-09-17T15:48:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 1 -- 172.21.15.64:0/3746576162 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 306851339 0 0) 0x7f46f8011280 con 0x7f46f800b1e0 2024-09-17T15:48:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient(hunting): my global_id is 5062 2024-09-17T15:48:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 1 -- 172.21.15.64:0/3746576162 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46f800ffe0 con 0x7f46f800b1e0 2024-09-17T15:48:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 1 -- 172.21.15.64:0/3746576162 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 638239442 0 0) 0x7f46f4003520 con 0x7f46f8007470 2024-09-17T15:48:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 1 -- 172.21.15.64:0/3746576162 >> v1:172.21.15.64:6791/0 conn(0x7f46f800b1e0 legacy=0x7f46f800d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 1 -- 172.21.15.64:0/3746576162 >> v1:172.21.15.64:6790/0 conn(0x7f46f80037e0 legacy=0x7f46f8005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient: _finish_hunting 0 2024-09-17T15:48:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 1 monclient: found mon.noname-f 2024-09-17T15:48:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f4710fa5640 1 -- 172.21.15.64:0/3746576162 reap_dead start 2024-09-17T15:48:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.100:6789/0 2024-09-17T15:48:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 1 -- 172.21.15.64:0/3746576162 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f8012580 con 0x7f46f8007470 2024-09-17T15:48:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient: _finish_auth 0 2024-09-17T15:48:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f4712a08640 5 monclient: authenticate success, global_id 5104 2024-09-17T15:48:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f4712a08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 1 -- 172.21.15.64:0/3746576162 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4700001870 con 0x7f46f8007470 2024-09-17T15:48:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.809+0000 7f470affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.810+0000 7f470affd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:48:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.810+0000 7f470affd640 10 monclient: dump: 2024-09-17T15:48:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.810+0000 7f4712a08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.810+0000 7f470affd640 1 -- 172.21.15.64:0/3746576162 <== mon.1 v1:172.21.15.100:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4700001a80 con 0x7f46f8007470 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.810+0000 7f470affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.810+0000 7f470affd640 1 -- 172.21.15.64:0/3746576162 <== mon.1 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4700003930 con 0x7f46f8007470 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.810+0000 7f470affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.810+0000 7f470affd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.810+0000 7f470affd640 10 monclient: dump: 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:01.820 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.810+0000 7f4712a08640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.810+0000 7f4712a08640 4 set_mon_vals no callback set 2024-09-17T15:48:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.811+0000 7f4712a08640 10 monclient: shutdown 2024-09-17T15:48:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.811+0000 7f4712a08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.811+0000 7f4712a08640 1 -- 172.21.15.64:0/3746576162 >> v1:172.21.15.100:6789/0 conn(0x7f46f8007470 legacy=0x7f46f8009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.811+0000 7f4712a08640 1 -- 172.21.15.64:0/3746576162 shutdown_connections 2024-09-17T15:48:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.811+0000 7f4712a08640 1 -- 172.21.15.64:0/3746576162 >> 172.21.15.64:0/3746576162 conn(0x7f470c06daa0 msgr2=0x7f470c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.811+0000 7f4712a08640 1 -- 172.21.15.64:0/3746576162 shutdown_connections 2024-09-17T15:48:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.811+0000 7f4712a08640 1 -- 172.21.15.64:0/3746576162 wait complete. 2024-09-17T15:48:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.812+0000 7f4712a08640 10 monclient: build_initial_monmap 2024-09-17T15:48:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.812+0000 7f4712a08640 10 monclient: monmap: 2024-09-17T15:48:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:01.822 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.812+0000 7f4712a08640 1 Processor -- start 2024-09-17T15:48:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f4712a08640 1 -- start start 2024-09-17T15:48:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f4712a08640 10 monclient: init 2024-09-17T15:48:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f4712a08640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f4712a08640 10 monclient: _add_conns ranks=[6,8,1,7,4,3,5,2,0] 2024-09-17T15:48:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f4712a08640 10 monclient(hunting): picked mon.noname-d con 0x7f46f8007470 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f4712a08640 10 monclient(hunting): picked mon.noname-e con 0x7f470c0a4180 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f4712a08640 10 monclient(hunting): picked mon.noname-f con 0x7f470c071910 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f4712a08640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c0725e0 con 0x7f470c071910 2024-09-17T15:48:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f4712a08640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c0a45e0 con 0x7f46f8007470 2024-09-17T15:48:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f4712a08640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c071d60 con 0x7f470c0a4180 2024-09-17T15:48:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f4712a08640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f4712a08640 10 monclient(hunting): authenticate will time out at 879.489766s 2024-09-17T15:48:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f470b7fe640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f470c0a4180 0x7f470c071200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:34900/0 (socket says 172.21.15.64:34900) 2024-09-17T15:48:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f470bfff640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f46f8007470 0x7f470c070af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:39280/0 (socket says 172.21.15.64:39280) 2024-09-17T15:48:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.813+0000 7f470b7fe640 1 -- 172.21.15.64:0/1513079324 learned_addr learned my addr 172.21.15.64:0/1513079324 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:01.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.834+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 54294369 0 0) 0x7f470c071d60 con 0x7f470c0a4180 2024-09-17T15:48:01.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.834+0000 7f4708ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:48:01.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.834+0000 7f4708ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:01.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.834+0000 7f4708ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:01.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.834+0000 7f4708ff9640 10 monclient(hunting): my global_id is 4967 2024-09-17T15:48:01.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.834+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46f8010290 con 0x7f470c0a4180 2024-09-17T15:48:01.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 147923537 0 0) 0x7f46f8010290 con 0x7f470c0a4180 2024-09-17T15:48:01.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f470c071d60 con 0x7f470c0a4180 2024-09-17T15:48:01.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4700003930 con 0x7f470c0a4180 2024-09-17T15:48:01.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:01.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:48:01.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 10 monclient(hunting): dump: 2024-09-17T15:48:01.839 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:01.840 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:01.840 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:01.840 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:01.840 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:01.840 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:01.840 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:01.840 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:01.840 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:01.840 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:01.840 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:01.840 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:01.840 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1903308200 0 0) 0x7f470c0725e0 con 0x7f470c071910 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 10 monclient(hunting): my global_id is 5122 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46f8010290 con 0x7f470c071910 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3065394466 0 0) 0x7f470c071d60 con 0x7f470c0a4180 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 >> v1:172.21.15.64:6792/0 conn(0x7f46f8007470 legacy=0x7f470c070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 >> v1:172.21.15.100:6789/0 conn(0x7f470c071910 legacy=0x7f470c1dac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:01.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 10 monclient: _finish_hunting 0 2024-09-17T15:48:01.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 1 monclient: found mon.e 2024-09-17T15:48:01.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:01.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:01.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c0740a0 con 0x7f470c0a4180 2024-09-17T15:48:01.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 10 monclient: _finish_auth 0 2024-09-17T15:48:01.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:48:01.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4708ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:01.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4712a08640 5 monclient: authenticate success, global_id 4967 2024-09-17T15:48:01.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4712a08640 10 monclient: _renew_subs 2024-09-17T15:48:01.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4712a08640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:01.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4712a08640 1 -- 172.21.15.64:0/1513079324 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f470c074380 con 0x7f470c0a4180 2024-09-17T15:48:01.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4712a08640 10 monclient: _renew_subs 2024-09-17T15:48:01.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4712a08640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:01.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.835+0000 7f4712a08640 1 -- 172.21.15.64:0/1513079324 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f470c073140 con 0x7f470c0a4180 2024-09-17T15:48:01.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.836+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4700001cb0 con 0x7f470c0a4180 2024-09-17T15:48:01.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.836+0000 7f4708ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:01.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.836+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4700004140 con 0x7f470c0a4180 2024-09-17T15:48:01.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.836+0000 7f4708ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:01.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.836+0000 7f4708ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:01.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.836+0000 7f470a7fc640 4 set_mon_vals no callback set 2024-09-17T15:48:01.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.836+0000 7f4708ff9640 10 monclient: dump: 2024-09-17T15:48:01.844 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:01.844 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:01.844 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:01.844 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:01.844 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:01.844 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:01.844 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:01.844 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:01.844 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:01.845 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:01.845 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:01.845 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:01.845 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:01.845 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:01.845 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:01.845 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:01.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.837+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f47000042e0 con 0x7f470c0a4180 2024-09-17T15:48:01.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.837+0000 7f4712a08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:01.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.837+0000 7f4712a08640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:01.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.837+0000 7f4712a08640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:01.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.837+0000 7f4712a08640 1 -- 172.21.15.64:0/1513079324 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46ec005080 con 0x7f470c0a4180 2024-09-17T15:48:01.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.837+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(108..108 src has 1..108) ==== 3898+0+0 (unknown 2836509362 0 0) 0x7f470004ec10 con 0x7f470c0a4180 2024-09-17T15:48:01.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.838+0000 7f470bfff640 10 monclient: get_auth_request con 0x7f46f8041ab0 auth_method 0 2024-09-17T15:48:01.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.843+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 <== mon.8 v1:172.21.15.64:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f4700017740 con 0x7f470c0a4180 2024-09-17T15:48:01.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.843+0000 7f4708ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:01.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:01.843+0000 7f4708ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:02.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.108+0000 7f4712a08640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:48:02.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.108+0000 7f4712a08640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:02.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.108+0000 7f4712a08640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:02.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.108+0000 7f4712a08640 1 -- 172.21.15.64:0/1513079324 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f46ec005370 con 0x7f470c0a4180 2024-09-17T15:48:02.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.108+0000 7f4708ff9640 1 -- 172.21.15.64:0/1513079324 <== mon.8 v1:172.21.15.64:6793/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 2719734712) 0x7f47000209a0 con 0x7f470c0a4180 2024-09-17T15:48:02.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.108+0000 7f4708ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:02.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.108+0000 7f4708ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:02.112 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":16,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:48:02.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.110+0000 7f4712a08640 1 -- 172.21.15.64:0/1513079324 >> v1:172.21.15.100:6812/4210963484 conn(0x7f46f8041ab0 legacy=0x7f46f8043f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.110+0000 7f4712a08640 10 monclient: shutdown 2024-09-17T15:48:02.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.110+0000 7f4712a08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:02.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.110+0000 7f4712a08640 1 -- 172.21.15.64:0/1513079324 >> v1:172.21.15.64:6793/0 conn(0x7f470c0a4180 legacy=0x7f470c071200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.110+0000 7f4712a08640 1 -- 172.21.15.64:0/1513079324 shutdown_connections 2024-09-17T15:48:02.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.110+0000 7f4712a08640 1 -- 172.21.15.64:0/1513079324 >> 172.21.15.64:0/1513079324 conn(0x7f470c06daa0 msgr2=0x7f470c132630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:02.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.110+0000 7f4712a08640 1 -- 172.21.15.64:0/1513079324 shutdown_connections 2024-09-17T15:48:02.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.110+0000 7f4712a08640 1 -- 172.21.15.64:0/1513079324 wait complete. 2024-09-17T15:48:02.124 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-17T15:48:02.124 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-17T15:48:02.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:48:02.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.222+0000 7faa07efe640 10 monclient: get_monmap_and_config 2024-09-17T15:48:02.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.222+0000 7faa07efe640 10 monclient: build_initial_monmap 2024-09-17T15:48:02.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.222+0000 7faa07efe640 10 monclient: monmap: 2024-09-17T15:48:02.225 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:02.225 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.225 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:02.224507+0000 2024-09-17T15:48:02.226 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:02.224507+0000 2024-09-17T15:48:02.226 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:02.226 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.226 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:02.226 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:02.226 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:02.226 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:02.226 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:02.226 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:02.226 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:02.226 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:02.226 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:02.226 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.223+0000 7faa07efe640 1 Processor -- start 2024-09-17T15:48:02.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.223+0000 7faa07efe640 1 -- start start 2024-09-17T15:48:02.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.223+0000 7faa07efe640 10 monclient: init 2024-09-17T15:48:02.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.223+0000 7faa07efe640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:02.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.223+0000 7faa07efe640 10 monclient: _add_conns ranks=[6,8,7,1,5,2,3,4,0] 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.223+0000 7faa07efe640 10 monclient(hunting): picked mon.noname-d con 0x7faa000a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.223+0000 7faa07efe640 10 monclient(hunting): picked mon.noname-e con 0x7faa00135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.223+0000 7faa07efe640 10 monclient(hunting): picked mon.noname-i con 0x7faa00135f70 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.223+0000 7faa07efe640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa0009d6f0 con 0x7faa000a5520 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.223+0000 7faa07efe640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa0009d9d0 con 0x7faa00135f70 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.223+0000 7faa07efe640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa0009dcb0 con 0x7faa00135b50 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.223+0000 7faa07efe640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.223+0000 7faa07efe640 10 monclient(hunting): authenticate will time out at 879.899991s 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa05c9a640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7faa000a5520 0x7faa000a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:39294/0 (socket says 172.21.15.64:39294) 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa05499640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7faa00135b50 0x7faa0013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:34902/0 (socket says 172.21.15.64:34902) 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa05c9a640 1 -- 172.21.15.64:0/3408991489 learned_addr learned my addr 172.21.15.64:0/3408991489 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 -- 172.21.15.64:0/3408991489 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1396130204 0 0) 0x7faa0009d6f0 con 0x7faa000a5520 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient(hunting): my global_id is 114108 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 -- 172.21.15.64:0/3408991489 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9ec003520 con 0x7faa000a5520 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 -- 172.21.15.64:0/3408991489 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4279473347 0 0) 0x7faa0009dcb0 con 0x7faa00135b50 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient(hunting): my global_id is 4976 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 -- 172.21.15.64:0/3408991489 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa0009d6f0 con 0x7faa00135b50 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 -- 172.21.15.64:0/3408991489 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3490592883 0 0) 0x7faa0009d9d0 con 0x7faa00135f70 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient(hunting): my global_id is 4948 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 -- 172.21.15.64:0/3408991489 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa0009dcb0 con 0x7faa00135f70 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 -- 172.21.15.64:0/3408991489 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2146629520 0 0) 0x7fa9ec003520 con 0x7faa000a5520 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 -- 172.21.15.64:0/3408991489 >> v1:172.21.15.64:6793/0 conn(0x7faa00135b50 legacy=0x7faa0013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 -- 172.21.15.64:0/3408991489 >> v1:172.21.15.100:6792/0 conn(0x7faa00135f70 legacy=0x7faa001415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient: _finish_hunting 0 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 monclient: found mon.noname-d 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 -- 172.21.15.64:0/3408991489 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa0009ccd0 con 0x7faa000a5520 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient: _finish_auth 0 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient: _check_auth_tickets 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 -- 172.21.15.64:0/3408991489 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa9fc001830 con 0x7faa000a5520 2024-09-17T15:48:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa07efe640 5 monclient: authenticate success, global_id 114108 2024-09-17T15:48:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa07efe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:48:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient: dump: 2024-09-17T15:48:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa07efe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 -- 172.21.15.64:0/3408991489 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa9fc0019d0 con 0x7faa000a5520 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 1 -- 172.21.15.64:0/3408991489 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa9fc004be0 con 0x7faa000a5520 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa07efe640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa07efe640 4 set_mon_vals no callback set 2024-09-17T15:48:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.224+0000 7faa04c98640 10 monclient: dump: 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.239 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.240 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.240 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.240 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.240 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.225+0000 7faa07efe640 10 monclient: shutdown 2024-09-17T15:48:02.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.225+0000 7faa07efe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:02.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.225+0000 7faa07efe640 1 -- 172.21.15.64:0/3408991489 >> v1:172.21.15.64:6792/0 conn(0x7faa000a5520 legacy=0x7faa000a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.225+0000 7faa07efe640 1 -- 172.21.15.64:0/3408991489 shutdown_connections 2024-09-17T15:48:02.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.225+0000 7faa07efe640 1 -- 172.21.15.64:0/3408991489 >> 172.21.15.64:0/3408991489 conn(0x7faa0006db30 msgr2=0x7faa000ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:02.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.225+0000 7faa07efe640 1 -- 172.21.15.64:0/3408991489 shutdown_connections 2024-09-17T15:48:02.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.225+0000 7faa07efe640 1 -- 172.21.15.64:0/3408991489 wait complete. 2024-09-17T15:48:02.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.225+0000 7faa07efe640 10 monclient: build_initial_monmap 2024-09-17T15:48:02.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.225+0000 7faa07efe640 10 monclient: monmap: 2024-09-17T15:48:02.241 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:02.241 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:02.241 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:02.241 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:02.241 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:02.241 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:02.241 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:02.241 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:02.241 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:02.241 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:02.241 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:02.241 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:02.241 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa07efe640 1 Processor -- start 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa07efe640 1 -- start start 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa07efe640 10 monclient: init 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa07efe640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa07efe640 10 monclient: _add_conns ranks=[7,4,1,3,2,0,8,6,5] 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa07efe640 10 monclient(hunting): picked mon.noname-i con 0x7faa000a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa07efe640 10 monclient(hunting): picked mon.noname-c con 0x7faa00135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa07efe640 10 monclient(hunting): picked mon.noname-f con 0x7faa00135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa07efe640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa000a0140 con 0x7faa00135f70 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa07efe640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa000a4320 con 0x7faa00135b50 2024-09-17T15:48:02.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa07efe640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa000a0420 con 0x7faa000a5520 2024-09-17T15:48:02.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa07efe640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:02.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa07efe640 10 monclient(hunting): authenticate will time out at 879.902764s 2024-09-17T15:48:02.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa05499640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7faa00135b50 0x7faa001321f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:50290/0 (socket says 172.21.15.64:50290) 2024-09-17T15:48:02.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.226+0000 7faa05499640 1 -- 172.21.15.64:0/205487657 learned_addr learned my addr 172.21.15.64:0/205487657 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:02.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 121360479 0 0) 0x7faa000a4320 con 0x7faa00135b50 2024-09-17T15:48:02.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): my global_id is 5071 2024-09-17T15:48:02.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9e4003520 con 0x7faa00135b50 2024-09-17T15:48:02.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2324693467 0 0) 0x7faa000a0140 con 0x7faa00135f70 2024-09-17T15:48:02.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): my global_id is 5131 2024-09-17T15:48:02.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa000a4320 con 0x7faa00135f70 2024-09-17T15:48:02.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771360427 0 0) 0x7faa000a0420 con 0x7faa000a5520 2024-09-17T15:48:02.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): my global_id is 4957 2024-09-17T15:48:02.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa000a0140 con 0x7faa000a5520 2024-09-17T15:48:02.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 530383632 0 0) 0x7fa9e4003520 con 0x7faa00135b50 2024-09-17T15:48:02.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa000a0420 con 0x7faa00135b50 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa9f4001690 con 0x7faa00135b50 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): dump: 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.245 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2982372109 0 0) 0x7faa000a0420 con 0x7faa00135b50 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 >> v1:172.21.15.100:6792/0 conn(0x7faa000a5520 legacy=0x7faa00131ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 >> v1:172.21.15.100:6789/0 conn(0x7faa00135f70 legacy=0x7faa001d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 monclient: found mon.c 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:02.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa00134a90 con 0x7faa00135b50 2024-09-17T15:48:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient: _finish_auth 0 2024-09-17T15:48:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:48:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7faa07efe640 5 monclient: authenticate success, global_id 5071 2024-09-17T15:48:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7faa07efe640 10 monclient: _renew_subs 2024-09-17T15:48:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7faa07efe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7faa07efe640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faa001346f0 con 0x7faa00135b50 2024-09-17T15:48:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7faa07efe640 10 monclient: _renew_subs 2024-09-17T15:48:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7faa07efe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7faa07efe640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7faa0009fe60 con 0x7faa00135b50 2024-09-17T15:48:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa9f4004a90 con 0x7faa00135b50 2024-09-17T15:48:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa9f4004f80 con 0x7faa00135b50 2024-09-17T15:48:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f3fff640 4 set_mon_vals no callback set 2024-09-17T15:48:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.227+0000 7fa9f27fc640 10 monclient: dump: 2024-09-17T15:48:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.249 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.228+0000 7faa07efe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.228+0000 7faa07efe640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa9e80065c0 con 0x7fa9e8002860 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.228+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fa9f4011860 con 0x7faa00135b50 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.228+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(108..108 src has 1..108) ==== 3898+0+0 (unknown 2836509362 0 0) 0x7fa9f404ed60 con 0x7faa00135b50 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.228+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4162210637 0 0) 0x7fa9e80065c0 con 0x7fa9e8002860 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.228+0000 7fa9f27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.228+0000 7fa9f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.228+0000 7fa9f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.228+0000 7fa9f27fc640 10 monclient(hunting): my global_id is 5013 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.228+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa000a0420 con 0x7fa9e8002860 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.228+0000 7faa0649b640 10 monclient: get_auth_request con 0x7fa9e40402e0 auth_method 0 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.229+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2982156328 0 0) 0x7faa000a0420 con 0x7fa9e8002860 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.229+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9e80065c0 con 0x7fa9e8002860 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.229+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa9fc004c20 con 0x7fa9e8002860 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.229+0000 7fa9f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.229+0000 7fa9f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:02.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.229+0000 7fa9f27fc640 10 monclient: dump: 2024-09-17T15:48:02.251 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.251 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.251 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.251 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.251 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.251 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.251 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.251 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.251 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.251 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.252 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.252 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.252 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.252 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.252 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.252 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.229+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 349095930 0 0) 0x7fa9e80065c0 con 0x7fa9e8002860 2024-09-17T15:48:02.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.229+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.64:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa9e800a160 con 0x7fa9e8002860 2024-09-17T15:48:02.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.229+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa9fc0015e0 con 0x7fa9e8002860 2024-09-17T15:48:02.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.229+0000 7fa9f27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:02.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.229+0000 7fa9f27fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:02.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.229+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 >> v1:172.21.15.64:6789/0 conn(0x7fa9e8002860 legacy=0x7fa9e8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.229+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 >> v1:172.21.15.64:6789/0 conn(0x7fa9e8002860 legacy=0x7fa9e8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:02.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.235+0000 7faa07efe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.235+0000 7faa07efe640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa9e8009e70 con 0x7fa9e8009130 2024-09-17T15:48:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1669839461 0 0) 0x7fa9e8009e70 con 0x7fa9e8009130 2024-09-17T15:48:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 10 monclient(hunting): my global_id is 5022 2024-09-17T15:48:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9e80065c0 con 0x7fa9e8009130 2024-09-17T15:48:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2816142074 0 0) 0x7fa9e80065c0 con 0x7fa9e8009130 2024-09-17T15:48:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9e8009e70 con 0x7fa9e8009130 2024-09-17T15:48:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa9fc005000 con 0x7fa9e8009130 2024-09-17T15:48:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 10 monclient: dump: 2024-09-17T15:48:02.256 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.256 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.256 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.256 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.256 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.256 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.256 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.256 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.256 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.256 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.256 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.256 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.256 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.257 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.257 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.257 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 522698986 0 0) 0x7fa9e8009e70 con 0x7fa9e8009130 2024-09-17T15:48:02.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 --> v1:172.21.15.64:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa9e800c410 con 0x7fa9e8009130 2024-09-17T15:48:02.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7713 (unknown 0 0 1019518800) 0x7fa9fc0051a0 con 0x7fa9e8009130 2024-09-17T15:48:02.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:02.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:02.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 >> v1:172.21.15.64:6789/0 conn(0x7fa9e8009130 legacy=0x7fa9e80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.236+0000 7fa9f27fc640 1 -- 172.21.15.64:0/205487657 >> v1:172.21.15.64:6789/0 conn(0x7fa9e8009130 legacy=0x7fa9e80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:02.258 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:02.258 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:02.258 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:02.258 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "leader", 2024-09-17T15:48:02.258 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:02.258 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 138890, 2024-09-17T15:48:02.258 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:02.258 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:02.258 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:02.258 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:02.258 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:02.258 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:02.258 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:48:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.259 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.260 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:02.261 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.262 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:02.263 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.264 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.265 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.266 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.267 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:02.268 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.269 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.269 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:02.272 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.272 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.272 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:02.272 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:02.275 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.276 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.276 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.276 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:02.276 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.276 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.276 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.276 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:02.276 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:02.276 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.276 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.276 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.277 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.278 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.280 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:02.280 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:02.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.237+0000 7faa07efe640 1 -- 172.21.15.64:0/205487657 >> v1:172.21.15.100:6812/4210963484 conn(0x7fa9e40402e0 legacy=0x7fa9e40427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.237+0000 7faa07efe640 10 monclient: shutdown 2024-09-17T15:48:02.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.237+0000 7faa07efe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:02.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.237+0000 7faa07efe640 1 -- 172.21.15.64:0/205487657 >> v1:172.21.15.64:6791/0 conn(0x7faa00135b50 legacy=0x7faa001321f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.237+0000 7faa0649b640 1 -- 172.21.15.64:0/205487657 reap_dead start 2024-09-17T15:48:02.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.237+0000 7faa0649b640 1 -- 172.21.15.64:0/205487657 reap_dead start 2024-09-17T15:48:02.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.237+0000 7faa07efe640 1 -- 172.21.15.64:0/205487657 shutdown_connections 2024-09-17T15:48:02.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.237+0000 7faa07efe640 1 -- 172.21.15.64:0/205487657 >> 172.21.15.64:0/205487657 conn(0x7faa0006db30 msgr2=0x7faa0012c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:02.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.237+0000 7faa07efe640 1 -- 172.21.15.64:0/205487657 shutdown_connections 2024-09-17T15:48:02.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.237+0000 7faa07efe640 1 -- 172.21.15.64:0/205487657 wait complete. 2024-09-17T15:48:02.282 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:02.282 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-17T15:48:02.282 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:48:02.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.381+0000 7f7c7807f640 10 monclient: get_monmap_and_config 2024-09-17T15:48:02.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.381+0000 7f7c7807f640 10 monclient: build_initial_monmap 2024-09-17T15:48:02.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.381+0000 7f7c7807f640 10 monclient: monmap: 2024-09-17T15:48:02.384 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:02.384 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.384 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:02.383343+0000 2024-09-17T15:48:02.384 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:02.383343+0000 2024-09-17T15:48:02.384 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:02.384 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.385 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:02.385 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:02.385 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:02.385 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:02.385 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:02.385 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:02.385 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:02.385 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:02.385 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:02.385 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c7807f640 1 Processor -- start 2024-09-17T15:48:02.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c7807f640 1 -- start start 2024-09-17T15:48:02.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c7807f640 10 monclient: init 2024-09-17T15:48:02.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c7807f640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:02.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c7807f640 10 monclient: _add_conns ranks=[0,5,3,6,8,4,1,2,7] 2024-09-17T15:48:02.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c7807f640 10 monclient(hunting): picked mon.noname-a con 0x7f7c700a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:02.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c7807f640 10 monclient(hunting): picked mon.noname-h con 0x7f7c70135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:02.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c7807f640 10 monclient(hunting): picked mon.noname-g con 0x7f7c70135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:02.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c7807f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c7009d6f0 con 0x7f7c700a5520 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c7807f640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c7009d9d0 con 0x7f7c70135f70 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c7807f640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c7009dcb0 con 0x7f7c70135b50 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c7807f640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c7807f640 10 monclient(hunting): authenticate will time out at 880.058822s 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c75e1b640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f7c700a5520 0x7f7c700a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:46996/0 (socket says 172.21.15.64:46996) 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.382+0000 7f7c75e1b640 1 -- 172.21.15.64:0/1177924161 learned_addr learned my addr 172.21.15.64:0/1177924161 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 1 -- 172.21.15.64:0/1177924161 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1158431045 0 0) 0x7f7c7009d6f0 con 0x7f7c700a5520 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient(hunting): my global_id is 5031 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 1 -- 172.21.15.64:0/1177924161 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c5c003520 con 0x7f7c700a5520 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 1 -- 172.21.15.64:0/1177924161 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2018440853 0 0) 0x7f7c5c003520 con 0x7f7c700a5520 2024-09-17T15:48:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 1 -- 172.21.15.64:0/1177924161 >> v1:172.21.15.100:6791/0 conn(0x7f7c70135b50 legacy=0x7f7c7013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 1 -- 172.21.15.64:0/1177924161 >> v1:172.21.15.100:6790/0 conn(0x7f7c70135f70 legacy=0x7f7c701415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient: _finish_hunting 0 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 1 monclient: found mon.noname-a 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 1 -- 172.21.15.64:0/1177924161 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c7009ccd0 con 0x7f7c700a5520 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient: _finish_auth 0 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient: _check_auth_tickets 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c7807f640 5 monclient: authenticate success, global_id 5031 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c7807f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 1 -- 172.21.15.64:0/1177924161 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7c6c001830 con 0x7f7c700a5520 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient: dump: 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c7807f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 1 -- 172.21.15.64:0/1177924161 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c6c001a50 con 0x7f7c700a5520 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c7807f640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 1 -- 172.21.15.64:0/1177924161 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7c6c004be0 con 0x7f7c700a5520 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c7807f640 4 set_mon_vals no callback set 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.383+0000 7f7c74e19640 10 monclient: dump: 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.398 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.398 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.398 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.398 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.398 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.398 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.398 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.398 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.398 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.398 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.398 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.398 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.398 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.399 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.384+0000 7f7c7807f640 10 monclient: shutdown 2024-09-17T15:48:02.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.384+0000 7f7c7807f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:02.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.384+0000 7f7c7807f640 1 -- 172.21.15.64:0/1177924161 >> v1:172.21.15.64:6789/0 conn(0x7f7c700a5520 legacy=0x7f7c700a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.384+0000 7f7c7807f640 1 -- 172.21.15.64:0/1177924161 shutdown_connections 2024-09-17T15:48:02.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.384+0000 7f7c7807f640 1 -- 172.21.15.64:0/1177924161 >> 172.21.15.64:0/1177924161 conn(0x7f7c7006db30 msgr2=0x7f7c700ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:02.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.384+0000 7f7c7807f640 1 -- 172.21.15.64:0/1177924161 shutdown_connections 2024-09-17T15:48:02.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.384+0000 7f7c7807f640 1 -- 172.21.15.64:0/1177924161 wait complete. 2024-09-17T15:48:02.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.384+0000 7f7c7807f640 10 monclient: build_initial_monmap 2024-09-17T15:48:02.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.384+0000 7f7c7807f640 10 monclient: monmap: 2024-09-17T15:48:02.399 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:02.399 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:02.399 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:02.400 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:02.400 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:02.400 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:02.400 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:02.400 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:02.400 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:02.400 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:02.400 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:02.400 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:02.400 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:02.400 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:02.400 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:02.400 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7807f640 1 Processor -- start 2024-09-17T15:48:02.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7807f640 1 -- start start 2024-09-17T15:48:02.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7807f640 10 monclient: init 2024-09-17T15:48:02.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7807f640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:02.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7807f640 10 monclient: _add_conns ranks=[3,6,8,1,5,4,7,2,0] 2024-09-17T15:48:02.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7807f640 10 monclient(hunting): picked mon.noname-g con 0x7f7c700a5520 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:02.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7807f640 10 monclient(hunting): picked mon.noname-d con 0x7f7c70135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:02.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7807f640 10 monclient(hunting): picked mon.noname-e con 0x7f7c70135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:02.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7807f640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c7009ea60 con 0x7f7c700a5520 2024-09-17T15:48:02.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7807f640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c700a4320 con 0x7f7c70135b50 2024-09-17T15:48:02.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7807f640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c70134b70 con 0x7f7c70135f70 2024-09-17T15:48:02.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7807f640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:02.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7807f640 10 monclient(hunting): authenticate will time out at 880.061603s 2024-09-17T15:48:02.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7561a640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f7c70135b50 0x7f7c70131f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:39310/0 (socket says 172.21.15.64:39310) 2024-09-17T15:48:02.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c75e1b640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f7c700a5520 0x7f7c70131830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:46718/0 (socket says 172.21.15.64:46718) 2024-09-17T15:48:02.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7561a640 1 -- 172.21.15.64:0/940842377 learned_addr learned my addr 172.21.15.64:0/940842377 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:02.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c7661c640 1 --1- 172.21.15.64:0/940842377 >> v1:172.21.15.64:6793/0 conn(0x7f7c70135f70 0x7f7c701d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:34906/0 (socket says 172.21.15.64:34906) 2024-09-17T15:48:02.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3608200348 0 0) 0x7f7c70134b70 con 0x7f7c70135f70 2024-09-17T15:48:02.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c667fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c667fc640 10 monclient(hunting): my global_id is 4985 2024-09-17T15:48:02.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c54003550 con 0x7f7c70135f70 2024-09-17T15:48:02.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.385+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1749951635 0 0) 0x7f7c700a4320 con 0x7f7c70135b50 2024-09-17T15:48:02.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): my global_id is 114117 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c70134b70 con 0x7f7c70135b50 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 220190715 0 0) 0x7f7c7009ea60 con 0x7f7c700a5520 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): my global_id is 114114 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c700a4320 con 0x7f7c700a5520 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3221222207 0 0) 0x7f7c54003550 con 0x7f7c70135f70 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c7009ea60 con 0x7f7c70135f70 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7c68001d70 con 0x7f7c70135f70 2024-09-17T15:48:02.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:02.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:48:02.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): dump: 2024-09-17T15:48:02.404 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.404 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.404 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.404 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.404 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.404 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.404 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.404 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.404 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.404 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2335235670 0 0) 0x7f7c70134b70 con 0x7f7c70135b50 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c54003550 con 0x7f7c70135b50 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7c60001a90 con 0x7f7c70135b50 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): dump: 2024-09-17T15:48:02.405 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.406 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.406 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.406 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.406 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.406 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.406 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.406 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.406 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.406 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.406 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.406 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.406 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.406 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.407 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.407 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 963564059 0 0) 0x7f7c700a4320 con 0x7f7c700a5520 2024-09-17T15:48:02.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c70134b70 con 0x7f7c700a5520 2024-09-17T15:48:02.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7c6c004b20 con 0x7f7c700a5520 2024-09-17T15:48:02.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:02.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:02.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient(hunting): dump: 2024-09-17T15:48:02.407 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.407 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.407 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.407 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 659647613 0 0) 0x7f7c7009ea60 con 0x7f7c70135f70 2024-09-17T15:48:02.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 >> v1:172.21.15.64:6792/0 conn(0x7f7c70135b50 legacy=0x7f7c70131f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 >> v1:172.21.15.100:6790/0 conn(0x7f7c700a5520 legacy=0x7f7c70131830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:02.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 monclient: found mon.e 2024-09-17T15:48:02.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:02.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:02.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c70132650 con 0x7f7c70135f70 2024-09-17T15:48:02.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient: _finish_auth 0 2024-09-17T15:48:02.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient: _check_auth_tickets 2024-09-17T15:48:02.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:02.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c7807f640 5 monclient: authenticate success, global_id 4985 2024-09-17T15:48:02.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c7807f640 10 monclient: _renew_subs 2024-09-17T15:48:02.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c7807f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c7807f640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7c700a06e0 con 0x7f7c70135f70 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c7807f640 10 monclient: _renew_subs 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c7807f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c7807f640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f7c700a4f00 con 0x7f7c70135f70 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c68002070 con 0x7f7c70135f70 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7c68003f80 con 0x7f7c70135f70 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c67fff640 4 set_mon_vals no callback set 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.386+0000 7f7c667fc640 10 monclient: dump: 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.410 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.387+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f7c680041a0 con 0x7f7c70135f70 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.387+0000 7f7c7807f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.387+0000 7f7c7807f640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7c580065c0 con 0x7f7c58002860 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.387+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(108..108 src has 1..108) ==== 3898+0+0 (unknown 2836509362 0 0) 0x7f7c6804eff0 con 0x7f7c70135f70 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.387+0000 7f7c7561a640 10 monclient: get_auth_request con 0x7f7c54040c80 auth_method 0 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.387+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 691510866 0 0) 0x7f7c580065c0 con 0x7f7c58002860 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.387+0000 7f7c667fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.387+0000 7f7c667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.387+0000 7f7c667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.387+0000 7f7c667fc640 10 monclient(hunting): my global_id is 4835 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.387+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c7009ea60 con 0x7f7c58002860 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.387+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 385522251 0 0) 0x7f7c7009ea60 con 0x7f7c58002860 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.387+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c580065c0 con 0x7f7c58002860 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.388+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7c6c001b50 con 0x7f7c58002860 2024-09-17T15:48:02.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.388+0000 7f7c667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.388+0000 7f7c667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.388+0000 7f7c667fc640 10 monclient: dump: 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.413 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.414 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.414 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.414 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.414 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.388+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3705869850 0 0) 0x7f7c580065c0 con 0x7f7c58002860 2024-09-17T15:48:02.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.388+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c5800a160 con 0x7f7c58002860 2024-09-17T15:48:02.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.388+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7c6c007480 con 0x7f7c58002860 2024-09-17T15:48:02.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.388+0000 7f7c667fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:02.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.388+0000 7f7c667fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:02.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.388+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 >> v1:172.21.15.64:6790/0 conn(0x7f7c58002860 legacy=0x7f7c58004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.388+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 >> v1:172.21.15.64:6790/0 conn(0x7f7c58002860 legacy=0x7f7c58004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.394+0000 7f7c7807f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.394+0000 7f7c7807f640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7c58009e20 con 0x7f7c580090e0 2024-09-17T15:48:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1389344133 0 0) 0x7f7c58009e20 con 0x7f7c580090e0 2024-09-17T15:48:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 10 monclient(hunting): my global_id is 4844 2024-09-17T15:48:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c580065c0 con 0x7f7c580090e0 2024-09-17T15:48:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2106432493 0 0) 0x7f7c580065c0 con 0x7f7c580090e0 2024-09-17T15:48:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c58009e20 con 0x7f7c580090e0 2024-09-17T15:48:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7c6c0055f0 con 0x7f7c580090e0 2024-09-17T15:48:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:02.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 10 monclient: dump: 2024-09-17T15:48:02.417 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.417 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.417 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.417 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.417 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.417 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.417 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.417 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.417 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.417 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.418 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.418 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.418 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.418 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.418 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.418 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3242685659 0 0) 0x7f7c58009e20 con 0x7f7c580090e0 2024-09-17T15:48:02.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 --> v1:172.21.15.64:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7c5800c3f0 con 0x7f7c580090e0 2024-09-17T15:48:02.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 664881014) 0x7f7c6c007630 con 0x7f7c580090e0 2024-09-17T15:48:02.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:02.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:02.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 >> v1:172.21.15.64:6790/0 conn(0x7f7c580090e0 legacy=0x7f7c58009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.395+0000 7f7c667fc640 1 -- 172.21.15.64:0/940842377 >> v1:172.21.15.64:6790/0 conn(0x7f7c580090e0 legacy=0x7f7c58009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:02.419 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 139040, 2024-09-17T15:48:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:02.439 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:02.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.396+0000 7f7c7807f640 1 -- 172.21.15.64:0/940842377 >> v1:172.21.15.100:6812/4210963484 conn(0x7f7c54040c80 legacy=0x7f7c54043140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.396+0000 7f7c7807f640 10 monclient: shutdown 2024-09-17T15:48:02.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.396+0000 7f7c7807f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:02.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.396+0000 7f7c7807f640 1 -- 172.21.15.64:0/940842377 >> v1:172.21.15.64:6793/0 conn(0x7f7c70135f70 legacy=0x7f7c701d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.396+0000 7f7c7661c640 1 -- 172.21.15.64:0/940842377 reap_dead start 2024-09-17T15:48:02.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.396+0000 7f7c7661c640 1 -- 172.21.15.64:0/940842377 reap_dead start 2024-09-17T15:48:02.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.396+0000 7f7c7807f640 1 -- 172.21.15.64:0/940842377 shutdown_connections 2024-09-17T15:48:02.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.396+0000 7f7c7807f640 1 -- 172.21.15.64:0/940842377 >> 172.21.15.64:0/940842377 conn(0x7f7c7006db30 msgr2=0x7f7c7012c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:02.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.396+0000 7f7c7807f640 1 -- 172.21.15.64:0/940842377 shutdown_connections 2024-09-17T15:48:02.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.396+0000 7f7c7807f640 1 -- 172.21.15.64:0/940842377 wait complete. 2024-09-17T15:48:02.441 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:02.441 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-17T15:48:02.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:48:02.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.540+0000 7fe034c33640 10 monclient: get_monmap_and_config 2024-09-17T15:48:02.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.540+0000 7fe034c33640 10 monclient: build_initial_monmap 2024-09-17T15:48:02.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.540+0000 7fe034c33640 10 monclient: monmap: 2024-09-17T15:48:02.543 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:02.543 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.543 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:02.542093+0000 2024-09-17T15:48:02.543 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:02.542093+0000 2024-09-17T15:48:02.543 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:02.543 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.543 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:02.543 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:02.544 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:02.544 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:02.544 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:02.544 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:02.544 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:02.544 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:02.544 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:02.544 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.540+0000 7fe034c33640 1 Processor -- start 2024-09-17T15:48:02.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.540+0000 7fe034c33640 1 -- start start 2024-09-17T15:48:02.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.540+0000 7fe034c33640 10 monclient: init 2024-09-17T15:48:02.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.541+0000 7fe034c33640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:02.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.541+0000 7fe034c33640 10 monclient: _add_conns ranks=[7,3,5,8,2,1,4,6,0] 2024-09-17T15:48:02.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.541+0000 7fe034c33640 10 monclient(hunting): picked mon.noname-i con 0x7fe0300a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:02.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.541+0000 7fe034c33640 10 monclient(hunting): picked mon.noname-g con 0x7fe030135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:02.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.541+0000 7fe034c33640 10 monclient(hunting): picked mon.noname-h con 0x7fe030135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:02.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.541+0000 7fe034c33640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe03009d6f0 con 0x7fe030135b50 2024-09-17T15:48:02.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.541+0000 7fe034c33640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe03009d9d0 con 0x7fe030135f70 2024-09-17T15:48:02.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.541+0000 7fe034c33640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe03009dcb0 con 0x7fe0300a5520 2024-09-17T15:48:02.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.541+0000 7fe034c33640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:02.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.541+0000 7fe034c33640 10 monclient(hunting): authenticate will time out at 880.217614s 2024-09-17T15:48:02.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.541+0000 7fe02ed9d640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7fe030135f70 0x7fe0301415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:42396/0 (socket says 172.21.15.64:42396) 2024-09-17T15:48:02.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.541+0000 7fe02ed9d640 1 -- 172.21.15.64:0/3272328181 learned_addr learned my addr 172.21.15.64:0/3272328181 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:02.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 -- 172.21.15.64:0/3272328181 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1267907692 0 0) 0x7fe03009d9d0 con 0x7fe030135f70 2024-09-17T15:48:02.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient(hunting): my global_id is 114116 2024-09-17T15:48:02.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 -- 172.21.15.64:0/3272328181 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe01c003520 con 0x7fe030135f70 2024-09-17T15:48:02.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 -- 172.21.15.64:0/3272328181 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2739817709 0 0) 0x7fe03009dcb0 con 0x7fe0300a5520 2024-09-17T15:48:02.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient(hunting): my global_id is 4966 2024-09-17T15:48:02.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 -- 172.21.15.64:0/3272328181 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe03009d9d0 con 0x7fe0300a5520 2024-09-17T15:48:02.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 -- 172.21.15.64:0/3272328181 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3738877661 0 0) 0x7fe03009d6f0 con 0x7fe030135b50 2024-09-17T15:48:02.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient(hunting): my global_id is 114123 2024-09-17T15:48:02.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 -- 172.21.15.64:0/3272328181 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe03009dcb0 con 0x7fe030135b50 2024-09-17T15:48:02.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 -- 172.21.15.64:0/3272328181 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1751214036 0 0) 0x7fe01c003520 con 0x7fe030135f70 2024-09-17T15:48:02.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 -- 172.21.15.64:0/3272328181 >> v1:172.21.15.100:6792/0 conn(0x7fe0300a5520 legacy=0x7fe0300a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 -- 172.21.15.64:0/3272328181 >> v1:172.21.15.100:6790/0 conn(0x7fe030135b50 legacy=0x7fe03013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient: _finish_hunting 0 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 monclient: found mon.noname-h 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.100:6791/0 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 -- 172.21.15.64:0/3272328181 --> v1:172.21.15.100:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03009ccd0 con 0x7fe030135f70 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient: _finish_auth 0 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 -- 172.21.15.64:0/3272328181 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe028001830 con 0x7fe030135f70 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe034c33640 5 monclient: authenticate success, global_id 114116 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe034c33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-17T15:48:02.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient: dump: 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.556 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.557 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.557 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe034c33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:02.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 -- 172.21.15.64:0/3272328181 <== mon.5 v1:172.21.15.100:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe028001a50 con 0x7fe030135f70 2024-09-17T15:48:02.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:02.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe034c33640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:02.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 1 -- 172.21.15.64:0/3272328181 <== mon.5 v1:172.21.15.100:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe028004be0 con 0x7fe030135f70 2024-09-17T15:48:02.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe034c33640 4 set_mon_vals no callback set 2024-09-17T15:48:02.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:48:02.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.542+0000 7fe02d59a640 10 monclient: dump: 2024-09-17T15:48:02.557 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.557 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.558 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.558 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.558 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.558 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.558 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.558 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.558 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.558 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.558 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.558 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.558 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.558 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.558 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.559 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.543+0000 7fe034c33640 10 monclient: shutdown 2024-09-17T15:48:02.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.543+0000 7fe034c33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:02.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.543+0000 7fe034c33640 1 -- 172.21.15.64:0/3272328181 >> v1:172.21.15.100:6791/0 conn(0x7fe030135f70 legacy=0x7fe0301415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.543+0000 7fe034c33640 1 -- 172.21.15.64:0/3272328181 shutdown_connections 2024-09-17T15:48:02.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.543+0000 7fe034c33640 1 -- 172.21.15.64:0/3272328181 >> 172.21.15.64:0/3272328181 conn(0x7fe03006db30 msgr2=0x7fe0300ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:02.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.543+0000 7fe034c33640 1 -- 172.21.15.64:0/3272328181 shutdown_connections 2024-09-17T15:48:02.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.543+0000 7fe034c33640 1 -- 172.21.15.64:0/3272328181 wait complete. 2024-09-17T15:48:02.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.543+0000 7fe034c33640 10 monclient: build_initial_monmap 2024-09-17T15:48:02.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.543+0000 7fe034c33640 10 monclient: monmap: 2024-09-17T15:48:02.559 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:02.559 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:02.559 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe034c33640 1 Processor -- start 2024-09-17T15:48:02.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe034c33640 1 -- start start 2024-09-17T15:48:02.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe034c33640 10 monclient: init 2024-09-17T15:48:02.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe034c33640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:02.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe034c33640 10 monclient: _add_conns ranks=[2,0,1,7,3,4,6,5,8] 2024-09-17T15:48:02.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe034c33640 10 monclient(hunting): picked mon.noname-b con 0x7fe0300a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:02.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe034c33640 10 monclient(hunting): picked mon.noname-a con 0x7fe030135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:02.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe034c33640 10 monclient(hunting): picked mon.noname-f con 0x7fe030135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:02.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe034c33640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0300a6420 con 0x7fe030135b50 2024-09-17T15:48:02.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe034c33640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe03009fdf0 con 0x7fe030135f70 2024-09-17T15:48:02.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe034c33640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0301350e0 con 0x7fe0300a5520 2024-09-17T15:48:02.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe034c33640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:02.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe034c33640 10 monclient(hunting): authenticate will time out at 880.220761s 2024-09-17T15:48:02.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe02e59c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe0300a5520 0x7fe0300a4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:33944/0 (socket says 172.21.15.64:33944) 2024-09-17T15:48:02.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe02dd9b640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fe030135b50 0x7fe0301d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:46998/0 (socket says 172.21.15.64:46998) 2024-09-17T15:48:02.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.544+0000 7fe02e59c640 1 -- 172.21.15.64:0/425872153 learned_addr learned my addr 172.21.15.64:0/425872153 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:02.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2662011677 0 0) 0x7fe03009dcb0 con 0x7fe030135b50 2024-09-17T15:48:02.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): my global_id is 5040 2024-09-17T15:48:02.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe010003550 con 0x7fe030135b50 2024-09-17T15:48:02.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3794008522 0 0) 0x7fe0301350e0 con 0x7fe0300a5520 2024-09-17T15:48:02.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): my global_id is 4853 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe03009dcb0 con 0x7fe0300a5520 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1383906018 0 0) 0x7fe03009ccd0 con 0x7fe030135f70 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): my global_id is 5140 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0301350e0 con 0x7fe030135f70 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4255432509 0 0) 0x7fe03009dcb0 con 0x7fe0300a5520 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe03009ccd0 con 0x7fe0300a5520 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2583958228 0 0) 0x7fe010003550 con 0x7fe030135b50 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe03009dcb0 con 0x7fe030135b50 2024-09-17T15:48:02.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1079328671 0 0) 0x7fe0301350e0 con 0x7fe030135f70 2024-09-17T15:48:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe010003550 con 0x7fe030135f70 2024-09-17T15:48:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe018001e10 con 0x7fe0300a5520 2024-09-17T15:48:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): dump: 2024-09-17T15:48:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.564 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.565 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.565 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.565 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.565 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.565 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.565 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.565 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.565 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:02.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe020001a90 con 0x7fe030135b50 2024-09-17T15:48:02.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:02.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:02.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): dump: 2024-09-17T15:48:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.566 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe028001040 con 0x7fe030135f70 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient(hunting): dump: 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.567 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.568 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.568 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.568 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.568 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.568 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.568 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.568 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.568 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3438796494 0 0) 0x7fe03009ccd0 con 0x7fe0300a5520 2024-09-17T15:48:02.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 >> v1:172.21.15.100:6789/0 conn(0x7fe030135f70 legacy=0x7fe0301d62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 >> v1:172.21.15.64:6789/0 conn(0x7fe030135b50 legacy=0x7fe0301d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:48:02.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 monclient: found mon.b 2024-09-17T15:48:02.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:02.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:02.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03009f820 con 0x7fe0300a5520 2024-09-17T15:48:02.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient: _finish_auth 0 2024-09-17T15:48:02.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:02.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:02.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe034c33640 5 monclient: authenticate success, global_id 4853 2024-09-17T15:48:02.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe034c33640 10 monclient: _renew_subs 2024-09-17T15:48:02.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe034c33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:02.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe034c33640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe0300a0690 con 0x7fe0300a5520 2024-09-17T15:48:02.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe034c33640 10 monclient: _renew_subs 2024-09-17T15:48:02.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe034c33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:02.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe034c33640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe0301cdd10 con 0x7fe0300a5520 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe018003c40 con 0x7fe0300a5520 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe0180041f0 con 0x7fe0300a5520 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe02cd99640 4 set_mon_vals no callback set 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.545+0000 7fe026ffd640 10 monclient: dump: 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.570 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.571 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.571 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.571 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.571 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.571 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.571 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.571 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.571 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.571 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.546+0000 7fe034c33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:02.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.546+0000 7fe034c33640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe0140065c0 con 0x7fe014002860 2024-09-17T15:48:02.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.546+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fe018004410 con 0x7fe0300a5520 2024-09-17T15:48:02.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(108..108 src has 1..108) ==== 3898+0+0 (unknown 2836509362 0 0) 0x7fe01804ef70 con 0x7fe0300a5520 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2303054606 0 0) 0x7fe0140065c0 con 0x7fe014002860 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 10 monclient(hunting): my global_id is 5080 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe03009ccd0 con 0x7fe014002860 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe02ed9d640 10 monclient: get_auth_request con 0x7fe010040c30 auth_method 0 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4222610310 0 0) 0x7fe03009ccd0 con 0x7fe014002860 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0140065c0 con 0x7fe014002860 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe020003e70 con 0x7fe014002860 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 10 monclient: dump: 2024-09-17T15:48:02.572 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.574 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.574 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 39444451 0 0) 0x7fe0140065c0 con 0x7fe014002860 2024-09-17T15:48:02.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe01400a160 con 0x7fe014002860 2024-09-17T15:48:02.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe020004090 con 0x7fe014002860 2024-09-17T15:48:02.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:02.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:02.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 >> v1:172.21.15.64:6791/0 conn(0x7fe014002860 legacy=0x7fe014004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.547+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 >> v1:172.21.15.64:6791/0 conn(0x7fe014002860 legacy=0x7fe014004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:02.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe034c33640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:02.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe034c33640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe014009e00 con 0x7fe0140090e0 2024-09-17T15:48:02.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2139588770 0 0) 0x7fe014009e00 con 0x7fe0140090e0 2024-09-17T15:48:02.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 10 monclient(hunting): my global_id is 5089 2024-09-17T15:48:02.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0140065c0 con 0x7fe0140090e0 2024-09-17T15:48:02.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3992973436 0 0) 0x7fe0140065c0 con 0x7fe0140090e0 2024-09-17T15:48:02.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe014009e00 con 0x7fe0140090e0 2024-09-17T15:48:02.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe020005b00 con 0x7fe0140090e0 2024-09-17T15:48:02.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:02.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 10 monclient: dump: 2024-09-17T15:48:02.577 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.577 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.577 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.577 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.577 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.577 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.577 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.577 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.578 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.578 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.578 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.578 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.578 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.578 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.578 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.578 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1497909514 0 0) 0x7fe014009e00 con 0x7fe0140090e0 2024-09-17T15:48:02.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 --> v1:172.21.15.64:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe01400c3a0 con 0x7fe0140090e0 2024-09-17T15:48:02.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3740716682) 0x7fe020007e30 con 0x7fe0140090e0 2024-09-17T15:48:02.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:02.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:02.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 >> v1:172.21.15.64:6791/0 conn(0x7fe0140090e0 legacy=0x7fe014009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.554+0000 7fe026ffd640 1 -- 172.21.15.64:0/425872153 >> v1:172.21.15.64:6791/0 conn(0x7fe0140090e0 legacy=0x7fe014009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:02.579 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 139212, 2024-09-17T15:48:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:48:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:02.599 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.555+0000 7fe034c33640 1 -- 172.21.15.64:0/425872153 >> v1:172.21.15.100:6812/4210963484 conn(0x7fe010040c30 legacy=0x7fe0100430f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.555+0000 7fe034c33640 10 monclient: shutdown 2024-09-17T15:48:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.555+0000 7fe034c33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.555+0000 7fe034c33640 1 -- 172.21.15.64:0/425872153 >> v1:172.21.15.64:6790/0 conn(0x7fe0300a5520 legacy=0x7fe0300a4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.555+0000 7fe02ed9d640 1 -- 172.21.15.64:0/425872153 reap_dead start 2024-09-17T15:48:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.555+0000 7fe02ed9d640 1 -- 172.21.15.64:0/425872153 reap_dead start 2024-09-17T15:48:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.555+0000 7fe034c33640 1 -- 172.21.15.64:0/425872153 shutdown_connections 2024-09-17T15:48:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.555+0000 7fe034c33640 1 -- 172.21.15.64:0/425872153 >> 172.21.15.64:0/425872153 conn(0x7fe03006db30 msgr2=0x7fe0300a7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.555+0000 7fe034c33640 1 -- 172.21.15.64:0/425872153 shutdown_connections 2024-09-17T15:48:02.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.555+0000 7fe034c33640 1 -- 172.21.15.64:0/425872153 wait complete. 2024-09-17T15:48:02.601 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:02.601 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-17T15:48:02.601 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-17T15:48:02.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.701+0000 7fc875850640 10 monclient: get_monmap_and_config 2024-09-17T15:48:02.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.701+0000 7fc875850640 10 monclient: build_initial_monmap 2024-09-17T15:48:02.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.701+0000 7fc875850640 10 monclient: monmap: 2024-09-17T15:48:02.704 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:02.704 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.704 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:02.703414+0000 2024-09-17T15:48:02.704 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:02.703414+0000 2024-09-17T15:48:02.704 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:02.704 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.705 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:02.705 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:02.705 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:02.705 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:02.705 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:02.705 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:02.705 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:02.705 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:02.705 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:02.705 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.702+0000 7fc875850640 1 Processor -- start 2024-09-17T15:48:02.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.702+0000 7fc875850640 1 -- start start 2024-09-17T15:48:02.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.702+0000 7fc875850640 10 monclient: init 2024-09-17T15:48:02.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.702+0000 7fc875850640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:02.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.702+0000 7fc875850640 10 monclient: _add_conns ranks=[2,5,7,1,0,4,3,6,8] 2024-09-17T15:48:02.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.702+0000 7fc875850640 10 monclient(hunting): picked mon.noname-b con 0x7fc8700a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:02.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.702+0000 7fc875850640 10 monclient(hunting): picked mon.noname-h con 0x7fc870135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:02.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.702+0000 7fc875850640 10 monclient(hunting): picked mon.noname-i con 0x7fc870135f70 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:02.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.702+0000 7fc875850640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc87009d6f0 con 0x7fc8700a5520 2024-09-17T15:48:02.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.702+0000 7fc875850640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc87009d9d0 con 0x7fc870135b50 2024-09-17T15:48:02.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.702+0000 7fc875850640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc87009dcb0 con 0x7fc870135f70 2024-09-17T15:48:02.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc875850640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:02.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc875850640 10 monclient(hunting): authenticate will time out at 880.379017s 2024-09-17T15:48:02.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86effd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc8700a5520 0x7fc8700a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:33954/0 (socket says 172.21.15.64:33954) 2024-09-17T15:48:02.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86effd640 1 -- 172.21.15.64:0/2834617955 learned_addr learned my addr 172.21.15.64:0/2834617955 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:02.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 1 -- 172.21.15.64:0/2834617955 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1229132948 0 0) 0x7fc87009d6f0 con 0x7fc8700a5520 2024-09-17T15:48:02.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient(hunting): my global_id is 4862 2024-09-17T15:48:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 1 -- 172.21.15.64:0/2834617955 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc85c003520 con 0x7fc8700a5520 2024-09-17T15:48:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 1 -- 172.21.15.64:0/2834617955 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3435439182 0 0) 0x7fc85c003520 con 0x7fc8700a5520 2024-09-17T15:48:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 1 -- 172.21.15.64:0/2834617955 >> v1:172.21.15.100:6792/0 conn(0x7fc870135f70 legacy=0x7fc8701415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 1 -- 172.21.15.64:0/2834617955 >> v1:172.21.15.100:6791/0 conn(0x7fc870135b50 legacy=0x7fc87013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 1 monclient: found mon.noname-b 2024-09-17T15:48:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:48:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 1 -- 172.21.15.64:0/2834617955 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87009ccd0 con 0x7fc8700a5520 2024-09-17T15:48:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:48:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:48:02.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:02.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc875850640 5 monclient: authenticate success, global_id 4862 2024-09-17T15:48:02.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 1 -- 172.21.15.64:0/2834617955 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc860001830 con 0x7fc8700a5520 2024-09-17T15:48:02.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc875850640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:02.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:02.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient: dump: 2024-09-17T15:48:02.716 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.716 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.716 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.716 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.716 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.716 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.716 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.717 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.717 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.717 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.717 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.717 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.717 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.717 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.717 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.717 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc875850640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:02.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 1 -- 172.21.15.64:0/2834617955 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc8600019d0 con 0x7fc8700a5520 2024-09-17T15:48:02.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:02.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc875850640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:02.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 1 -- 172.21.15.64:0/2834617955 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc860004be0 con 0x7fc8700a5520 2024-09-17T15:48:02.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc875850640 4 set_mon_vals no callback set 2024-09-17T15:48:02.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:02.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.703+0000 7fc86e7fc640 10 monclient: dump: 2024-09-17T15:48:02.718 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.718 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.718 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.718 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.718 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.718 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.718 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.718 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.719 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.719 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.719 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.719 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.719 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.719 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.719 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.719 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.704+0000 7fc875850640 10 monclient: shutdown 2024-09-17T15:48:02.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.704+0000 7fc875850640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:02.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.704+0000 7fc875850640 1 -- 172.21.15.64:0/2834617955 >> v1:172.21.15.64:6790/0 conn(0x7fc8700a5520 legacy=0x7fc8700a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.704+0000 7fc875850640 1 -- 172.21.15.64:0/2834617955 shutdown_connections 2024-09-17T15:48:02.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.704+0000 7fc875850640 1 -- 172.21.15.64:0/2834617955 >> 172.21.15.64:0/2834617955 conn(0x7fc87006db30 msgr2=0x7fc8700ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.704+0000 7fc875850640 1 -- 172.21.15.64:0/2834617955 shutdown_connections 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.704+0000 7fc875850640 1 -- 172.21.15.64:0/2834617955 wait complete. 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.704+0000 7fc875850640 10 monclient: build_initial_monmap 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.704+0000 7fc875850640 10 monclient: monmap: 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc875850640 1 Processor -- start 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc875850640 1 -- start start 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc875850640 10 monclient: init 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc875850640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc875850640 10 monclient: _add_conns ranks=[6,5,0,7,2,8,1,4,3] 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc875850640 10 monclient(hunting): picked mon.noname-d con 0x7fc8700a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc875850640 10 monclient(hunting): picked mon.noname-h con 0x7fc870135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:02.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc875850640 10 monclient(hunting): picked mon.noname-a con 0x7fc870135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:02.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc875850640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8701cb050 con 0x7fc870135f70 2024-09-17T15:48:02.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc875850640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc870135310 con 0x7fc870135b50 2024-09-17T15:48:02.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc875850640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc870134f50 con 0x7fc8700a5520 2024-09-17T15:48:02.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc875850640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:02.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc875850640 10 monclient(hunting): authenticate will time out at 880.381860s 2024-09-17T15:48:02.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc86effd640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fc8700a5520 0x7fc8701ca230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:39316/0 (socket says 172.21.15.64:39316) 2024-09-17T15:48:02.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.705+0000 7fc86effd640 1 -- 172.21.15.64:0/1452537663 learned_addr learned my addr 172.21.15.64:0/1452537663 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:02.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4092318594 0 0) 0x7fc870134f50 con 0x7fc8700a5520 2024-09-17T15:48:02.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): my global_id is 114126 2024-09-17T15:48:02.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc850003550 con 0x7fc8700a5520 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3580573942 0 0) 0x7fc8701cb050 con 0x7fc870135f70 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): my global_id is 5049 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc870134f50 con 0x7fc870135f70 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2715497174 0 0) 0x7fc870135310 con 0x7fc870135b50 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): my global_id is 114134 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8701cb050 con 0x7fc870135b50 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3286897849 0 0) 0x7fc850003550 con 0x7fc8700a5520 2024-09-17T15:48:02.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc870135310 con 0x7fc8700a5520 2024-09-17T15:48:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc8600047e0 con 0x7fc8700a5520 2024-09-17T15:48:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:48:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): dump: 2024-09-17T15:48:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3910320630 0 0) 0x7fc870134f50 con 0x7fc870135f70 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc850003550 con 0x7fc870135f70 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc868001cb0 con 0x7fc870135f70 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): dump: 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.726 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2349511613 0 0) 0x7fc8701cb050 con 0x7fc870135b50 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc870134f50 con 0x7fc870135b50 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc8580019d0 con 0x7fc870135b50 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient(hunting): dump: 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1293212971 0 0) 0x7fc870135310 con 0x7fc8700a5520 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 >> v1:172.21.15.100:6791/0 conn(0x7fc870135b50 legacy=0x7fc8701ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 >> v1:172.21.15.64:6789/0 conn(0x7fc870135f70 legacy=0x7fc8701d5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient: _finish_hunting 0 2024-09-17T15:48:02.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 monclient: found mon.d 2024-09-17T15:48:02.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:02.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:48:02.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87009fdb0 con 0x7fc8700a5520 2024-09-17T15:48:02.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient: _finish_auth 0 2024-09-17T15:48:02.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient: _check_auth_tickets 2024-09-17T15:48:02.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:02.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc875850640 5 monclient: authenticate success, global_id 114126 2024-09-17T15:48:02.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc875850640 10 monclient: _renew_subs 2024-09-17T15:48:02.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc875850640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:48:02.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc875850640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc8700a0650 con 0x7fc8700a5520 2024-09-17T15:48:02.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc875850640 10 monclient: _renew_subs 2024-09-17T15:48:02.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc875850640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc875850640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fc87009e470 con 0x7fc8700a5520 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc860004c30 con 0x7fc8700a5520 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc860001950 con 0x7fc8700a5520 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86dffb640 4 set_mon_vals no callback set 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.706+0000 7fc86cff9640 10 monclient: dump: 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.730 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.707+0000 7fc875850640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.707+0000 7fc875850640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc8540065c0 con 0x7fc854002860 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.707+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fc860011960 con 0x7fc8700a5520 2024-09-17T15:48:02.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.707+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(108..108 src has 1..108) ==== 3898+0+0 (unknown 2836509362 0 0) 0x7fc86004ecb0 con 0x7fc8700a5520 2024-09-17T15:48:02.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1981065811 0 0) 0x7fc8540065c0 con 0x7fc854002860 2024-09-17T15:48:02.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 10 monclient(hunting): my global_id is 114135 2024-09-17T15:48:02.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc870135310 con 0x7fc854002860 2024-09-17T15:48:02.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86f7fe640 10 monclient: get_auth_request con 0x7fc850040da0 auth_method 0 2024-09-17T15:48:02.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1860088782 0 0) 0x7fc870135310 con 0x7fc854002860 2024-09-17T15:48:02.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8540065c0 con 0x7fc854002860 2024-09-17T15:48:02.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc858003930 con 0x7fc854002860 2024-09-17T15:48:02.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:48:02.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 10 monclient: dump: 2024-09-17T15:48:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.734 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.734 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.734 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4286619475 0 0) 0x7fc8540065c0 con 0x7fc854002860 2024-09-17T15:48:02.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc85400a160 con 0x7fc854002860 2024-09-17T15:48:02.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc858005bb0 con 0x7fc854002860 2024-09-17T15:48:02.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:02.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:02.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 >> v1:172.21.15.64:6792/0 conn(0x7fc854002860 legacy=0x7fc854004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.708+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 >> v1:172.21.15.64:6792/0 conn(0x7fc854002860 legacy=0x7fc854004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:02.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc875850640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:02.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc875850640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc854009e50 con 0x7fc854009110 2024-09-17T15:48:02.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 857472114 0 0) 0x7fc854009e50 con 0x7fc854009110 2024-09-17T15:48:02.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 10 monclient(hunting): my global_id is 114144 2024-09-17T15:48:02.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8540065c0 con 0x7fc854009110 2024-09-17T15:48:02.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4214758304 0 0) 0x7fc8540065c0 con 0x7fc854009110 2024-09-17T15:48:02.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc854009e50 con 0x7fc854009110 2024-09-17T15:48:02.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc8580054a0 con 0x7fc854009110 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 10 monclient: dump: 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.737 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.738 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.738 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.738 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.738 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.738 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 583744857 0 0) 0x7fc854009e50 con 0x7fc854009110 2024-09-17T15:48:02.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.715+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 --> v1:172.21.15.64:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc85400c3f0 con 0x7fc854009110 2024-09-17T15:48:02.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 2331357517) 0x7fc858005510 con 0x7fc854009110 2024-09-17T15:48:02.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc86cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:02.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc86cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:02.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 >> v1:172.21.15.64:6792/0 conn(0x7fc854009110 legacy=0x7fc8540095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc86cff9640 1 -- 172.21.15.64:0/1452537663 >> v1:172.21.15.64:6792/0 conn(0x7fc854009110 legacy=0x7fc8540095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:02.739 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:02.739 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:02.739 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:02.739 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:02.739 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:02.739 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 5930, 2024-09-17T15:48:02.739 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:02.739 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:02.739 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:02.739 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:02.739 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.740 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.741 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.741 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.741 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.741 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.741 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.741 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.741 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.741 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.741 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:02.741 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:02.741 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.741 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.741 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:02.742 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.743 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:02.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.745 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:02.745 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:02.745 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.745 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.745 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.745 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:02.745 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.745 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.745 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.745 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:02.745 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:02.746 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.746 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.746 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.746 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:02.746 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:02.746 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.746 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.746 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.746 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:02.747 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.747 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.747 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:02.747 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:02.747 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.747 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.747 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.747 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:02.747 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:02.748 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.748 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.748 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.748 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:02.748 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.748 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.748 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.748 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:02.748 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:02.749 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.749 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.749 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.749 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:02.749 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:02.749 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.749 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.749 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.749 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:02.749 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.750 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.750 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.750 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:02.750 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:02.750 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.750 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.750 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.750 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:02.750 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:02.750 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.752 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:02.752 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:02.752 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.752 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.752 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.752 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:02.752 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.752 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.752 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.753 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:02.754 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.756 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:02.756 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:02.756 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.756 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.756 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.756 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:02.756 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.756 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.756 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.756 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:02.757 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.758 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.758 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:02.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.758 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:02.758 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:02.758 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:48:02.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.758 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.758 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:02.758 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:02.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc875850640 1 -- 172.21.15.64:0/1452537663 >> v1:172.21.15.100:6812/4210963484 conn(0x7fc850040da0 legacy=0x7fc850043260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc875850640 10 monclient: shutdown 2024-09-17T15:48:02.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc875850640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:02.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc875850640 1 -- 172.21.15.64:0/1452537663 >> v1:172.21.15.64:6792/0 conn(0x7fc8700a5520 legacy=0x7fc8701ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc86f7fe640 1 -- 172.21.15.64:0/1452537663 reap_dead start 2024-09-17T15:48:02.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc86f7fe640 1 -- 172.21.15.64:0/1452537663 reap_dead start 2024-09-17T15:48:02.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc875850640 1 -- 172.21.15.64:0/1452537663 shutdown_connections 2024-09-17T15:48:02.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc875850640 1 -- 172.21.15.64:0/1452537663 >> 172.21.15.64:0/1452537663 conn(0x7fc87006db30 msgr2=0x7fc87012c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:02.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc875850640 1 -- 172.21.15.64:0/1452537663 shutdown_connections 2024-09-17T15:48:02.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.716+0000 7fc875850640 1 -- 172.21.15.64:0/1452537663 wait complete. 2024-09-17T15:48:02.761 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:02.761 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-17T15:48:02.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-17T15:48:02.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.872+0000 7feb47d17640 10 monclient: get_monmap_and_config 2024-09-17T15:48:02.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.872+0000 7feb47d17640 10 monclient: build_initial_monmap 2024-09-17T15:48:02.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.872+0000 7feb47d17640 10 monclient: monmap: 2024-09-17T15:48:02.875 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:02.874774+0000 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:02.874774+0000 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:02.877 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.873+0000 7feb47d17640 1 Processor -- start 2024-09-17T15:48:02.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.873+0000 7feb47d17640 1 -- start start 2024-09-17T15:48:02.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.873+0000 7feb47d17640 10 monclient: init 2024-09-17T15:48:02.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb47d17640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:02.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb47d17640 10 monclient: _add_conns ranks=[0,5,6,7,3,4,8,2,1] 2024-09-17T15:48:02.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb47d17640 10 monclient(hunting): picked mon.noname-a con 0x7feb400a54b0 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:02.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb47d17640 10 monclient(hunting): picked mon.noname-h con 0x7feb40135c80 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:02.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb47d17640 10 monclient(hunting): picked mon.noname-d con 0x7feb401360a0 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:02.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb47d17640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb400aacb0 con 0x7feb400a54b0 2024-09-17T15:48:02.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb47d17640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb4009d780 con 0x7feb40135c80 2024-09-17T15:48:02.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb47d17640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb4009da60 con 0x7feb401360a0 2024-09-17T15:48:02.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb47d17640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:02.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb47d17640 10 monclient(hunting): authenticate will time out at 880.550447s 2024-09-17T15:48:02.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb45ab3640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7feb400a54b0 0x7feb400a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:47012/0 (socket says 172.21.15.64:47012) 2024-09-17T15:48:02.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb462b4640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7feb401360a0 0x7feb40141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:39360/0 (socket says 172.21.15.64:39360) 2024-09-17T15:48:02.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb45ab3640 1 -- 172.21.15.64:0/184938937 learned_addr learned my addr 172.21.15.64:0/184938937 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:02.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 1 -- 172.21.15.64:0/184938937 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2102026949 0 0) 0x7feb4009da60 con 0x7feb401360a0 2024-09-17T15:48:02.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 10 monclient(hunting): my global_id is 114153 2024-09-17T15:48:02.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 1 -- 172.21.15.64:0/184938937 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb2c003520 con 0x7feb401360a0 2024-09-17T15:48:02.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 1 -- 172.21.15.64:0/184938937 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 141298855 0 0) 0x7feb400aacb0 con 0x7feb400a54b0 2024-09-17T15:48:02.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 10 monclient(hunting): my global_id is 5058 2024-09-17T15:48:02.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 1 -- 172.21.15.64:0/184938937 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb4009da60 con 0x7feb400a54b0 2024-09-17T15:48:02.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 1 -- 172.21.15.64:0/184938937 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3183968451 0 0) 0x7feb4009d780 con 0x7feb40135c80 2024-09-17T15:48:02.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 10 monclient(hunting): my global_id is 114143 2024-09-17T15:48:02.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.874+0000 7feb44ab1640 1 -- 172.21.15.64:0/184938937 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb400aacb0 con 0x7feb40135c80 2024-09-17T15:48:02.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 1 -- 172.21.15.64:0/184938937 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2493474915 0 0) 0x7feb2c003520 con 0x7feb401360a0 2024-09-17T15:48:02.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 1 -- 172.21.15.64:0/184938937 >> v1:172.21.15.100:6791/0 conn(0x7feb40135c80 legacy=0x7feb4013df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 1 -- 172.21.15.64:0/184938937 >> v1:172.21.15.64:6789/0 conn(0x7feb400a54b0 legacy=0x7feb400a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 10 monclient: _finish_hunting 0 2024-09-17T15:48:02.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 1 monclient: found mon.noname-d 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 1 -- 172.21.15.64:0/184938937 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4009dd40 con 0x7feb401360a0 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 10 monclient: _finish_auth 0 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 10 monclient: _check_auth_tickets 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 1 -- 172.21.15.64:0/184938937 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7feb38001ae0 con 0x7feb401360a0 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 10 monclient: dump: 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb47d17640 5 monclient: authenticate success, global_id 114153 2024-09-17T15:48:02.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb47d17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 1 -- 172.21.15.64:0/184938937 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb38001c80 con 0x7feb401360a0 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 1 -- 172.21.15.64:0/184938937 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7feb38004b60 con 0x7feb401360a0 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb47d17640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb47d17640 4 set_mon_vals no callback set 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb44ab1640 10 monclient: dump: 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.893 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb47d17640 10 monclient: shutdown 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb47d17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb47d17640 1 -- 172.21.15.64:0/184938937 >> v1:172.21.15.64:6792/0 conn(0x7feb401360a0 legacy=0x7feb40141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb47d17640 1 -- 172.21.15.64:0/184938937 shutdown_connections 2024-09-17T15:48:02.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb47d17640 1 -- 172.21.15.64:0/184938937 >> 172.21.15.64:0/184938937 conn(0x7feb4006daa0 msgr2=0x7feb400ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:02.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb47d17640 1 -- 172.21.15.64:0/184938937 shutdown_connections 2024-09-17T15:48:02.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.875+0000 7feb47d17640 1 -- 172.21.15.64:0/184938937 wait complete. 2024-09-17T15:48:02.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.876+0000 7feb47d17640 10 monclient: build_initial_monmap 2024-09-17T15:48:02.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.876+0000 7feb47d17640 10 monclient: monmap: 2024-09-17T15:48:02.895 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:02.895 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:02.895 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:02.895 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:02.895 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:02.895 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:02.895 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:02.895 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.876+0000 7feb47d17640 1 Processor -- start 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.876+0000 7feb47d17640 1 -- start start 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.876+0000 7feb47d17640 10 monclient: init 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb47d17640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb47d17640 10 monclient: _add_conns ranks=[5,6,7,3,8,0,2,4,1] 2024-09-17T15:48:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb47d17640 10 monclient(hunting): picked mon.noname-h con 0x7feb400a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb47d17640 10 monclient(hunting): picked mon.noname-d con 0x7feb40135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb47d17640 10 monclient(hunting): picked mon.noname-i con 0x7feb401360a0 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb47d17640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb4009e4b0 con 0x7feb400a54b0 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb47d17640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb40132860 con 0x7feb40135c80 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb47d17640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb401351d0 con 0x7feb401360a0 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb47d17640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb47d17640 10 monclient(hunting): authenticate will time out at 880.553384s 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb452b2640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7feb40135c80 0x7feb40132150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:39370/0 (socket says 172.21.15.64:39370) 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb452b2640 1 -- 172.21.15.64:0/1217797848 learned_addr learned my addr 172.21.15.64:0/1217797848 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3811522316 0 0) 0x7feb40132860 con 0x7feb40135c80 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 10 monclient(hunting): my global_id is 114162 2024-09-17T15:48:02.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb24003520 con 0x7feb40135c80 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1295819920 0 0) 0x7feb4009e4b0 con 0x7feb400a54b0 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 10 monclient(hunting): my global_id is 114152 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb40132860 con 0x7feb400a54b0 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1127528643 0 0) 0x7feb401351d0 con 0x7feb401360a0 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 10 monclient(hunting): my global_id is 4984 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb4009e4b0 con 0x7feb401360a0 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.877+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2791736647 0 0) 0x7feb24003520 con 0x7feb40135c80 2024-09-17T15:48:02.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb401351d0 con 0x7feb40135c80 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7feb30001b30 con 0x7feb40135c80 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient(hunting): dump: 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.899 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.900 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.900 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.900 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.900 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.900 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.900 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:02.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3378752463 0 0) 0x7feb40132860 con 0x7feb400a54b0 2024-09-17T15:48:02.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb24003520 con 0x7feb400a54b0 2024-09-17T15:48:02.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 77270934 0 0) 0x7feb4009e4b0 con 0x7feb401360a0 2024-09-17T15:48:02.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb40132860 con 0x7feb401360a0 2024-09-17T15:48:02.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7feb3c001470 con 0x7feb400a54b0 2024-09-17T15:48:02.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:48:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient(hunting): dump: 2024-09-17T15:48:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.901 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7feb380019a0 con 0x7feb401360a0 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient(hunting): dump: 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.902 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2980975970 0 0) 0x7feb401351d0 con 0x7feb40135c80 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 >> v1:172.21.15.100:6792/0 conn(0x7feb401360a0 legacy=0x7feb401d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 >> v1:172.21.15.100:6791/0 conn(0x7feb400a54b0 legacy=0x7feb40131a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:02.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 monclient: found mon.d 2024-09-17T15:48:02.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:02.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:48:02.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4009ff70 con 0x7feb40135c80 2024-09-17T15:48:02.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient: _finish_auth 0 2024-09-17T15:48:02.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient: _check_auth_tickets 2024-09-17T15:48:02.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:02.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb47d17640 5 monclient: authenticate success, global_id 114162 2024-09-17T15:48:02.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb47d17640 10 monclient: _renew_subs 2024-09-17T15:48:02.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb47d17640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:48:02.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb47d17640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feb400a0250 con 0x7feb40135c80 2024-09-17T15:48:02.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb47d17640 10 monclient: _renew_subs 2024-09-17T15:48:02.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb47d17640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:48:02.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb47d17640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7feb4009f010 con 0x7feb40135c80 2024-09-17T15:48:02.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb300038c0 con 0x7feb40135c80 2024-09-17T15:48:02.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:02.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb37fff640 4 set_mon_vals no callback set 2024-09-17T15:48:02.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7feb30003e30 con 0x7feb40135c80 2024-09-17T15:48:02.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:48:02.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.878+0000 7feb367fc640 10 monclient: dump: 2024-09-17T15:48:02.905 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.905 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.905 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.906 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.906 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.906 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.906 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.906 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.906 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.906 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.906 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.906 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.906 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.906 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.906 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.906 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.879+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7feb30003fd0 con 0x7feb40135c80 2024-09-17T15:48:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.879+0000 7feb47d17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.879+0000 7feb47d17640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feb280065c0 con 0x7feb28002860 2024-09-17T15:48:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.879+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(109..109 src has 1..109) ==== 3898+0+0 (unknown 1074856076 0 0) 0x7feb3004eb90 con 0x7feb40135c80 2024-09-17T15:48:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.879+0000 7feb462b4640 10 monclient: get_auth_request con 0x7feb24040de0 auth_method 0 2024-09-17T15:48:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.879+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1178379451 0 0) 0x7feb280065c0 con 0x7feb28002860 2024-09-17T15:48:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.879+0000 7feb367fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.879+0000 7feb367fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.879+0000 7feb367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.879+0000 7feb367fc640 10 monclient(hunting): my global_id is 4994 2024-09-17T15:48:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.879+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb401351d0 con 0x7feb28002860 2024-09-17T15:48:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.879+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 560391441 0 0) 0x7feb401351d0 con 0x7feb28002860 2024-09-17T15:48:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.880+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb280065c0 con 0x7feb28002860 2024-09-17T15:48:02.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.880+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7feb3c004b60 con 0x7feb28002860 2024-09-17T15:48:02.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.880+0000 7feb367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.880+0000 7feb367fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:02.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.880+0000 7feb367fc640 10 monclient: dump: 2024-09-17T15:48:02.908 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.908 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.908 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.908 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.908 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.908 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.908 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.908 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.908 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.880+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 278115181 0 0) 0x7feb280065c0 con 0x7feb28002860 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.880+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.64:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb2800a160 con 0x7feb28002860 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.880+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7feb3c004f00 con 0x7feb28002860 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.880+0000 7feb367fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.880+0000 7feb367fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.880+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 >> v1:172.21.15.64:6793/0 conn(0x7feb28002860 legacy=0x7feb28004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.880+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 >> v1:172.21.15.64:6793/0 conn(0x7feb28002860 legacy=0x7feb28004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:02.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.887+0000 7feb47d17640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:02.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.887+0000 7feb47d17640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feb28009e20 con 0x7feb280090e0 2024-09-17T15:48:02.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3913904545 0 0) 0x7feb28009e20 con 0x7feb280090e0 2024-09-17T15:48:02.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:02.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:02.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:02.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 10 monclient(hunting): my global_id is 5003 2024-09-17T15:48:02.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb280065c0 con 0x7feb280090e0 2024-09-17T15:48:02.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1672273658 0 0) 0x7feb280065c0 con 0x7feb280090e0 2024-09-17T15:48:02.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb28009e20 con 0x7feb280090e0 2024-09-17T15:48:02.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7feb3c006e60 con 0x7feb280090e0 2024-09-17T15:48:02.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:02.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:02.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 10 monclient: dump: 2024-09-17T15:48:02.912 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:02.912 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:02.912 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.912 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:02.912 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3513142326 0 0) 0x7feb28009e20 con 0x7feb280090e0 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 --> v1:172.21.15.64:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feb2800c3f0 con 0x7feb280090e0 2024-09-17T15:48:02.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3057899599) 0x7feb3c0067c0 con 0x7feb280090e0 2024-09-17T15:48:02.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:02.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:02.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 >> v1:172.21.15.64:6793/0 conn(0x7feb280090e0 legacy=0x7feb28009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.888+0000 7feb367fc640 1 -- 172.21.15.64:0/1217797848 >> v1:172.21.15.64:6793/0 conn(0x7feb280090e0 legacy=0x7feb28009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:02.914 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:02.914 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:02.914 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:02.914 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:02.914 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:02.914 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 139536, 2024-09-17T15:48:02.915 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:02.915 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:02.915 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:02.915 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:02.915 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:02.915 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:02.915 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:02.915 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:02.915 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:02.915 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:02.915 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.915 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:48:02.915 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.916 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.917 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:02.918 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:02.919 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:02.920 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:02.921 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:02.923 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.924 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.925 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.926 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.927 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.928 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.929 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:02.930 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.931 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:02.932 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:48:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:02.933 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:02.933 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:02.935 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36413]: starting. 2024-09-17T15:48:02.935 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36413]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36412 2024-09-17T15:48:02.935 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36413]: created object 0... 2024-09-17T15:48:02.935 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36413]: created object 25... 2024-09-17T15:48:02.935 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36413]: created object 49... 2024-09-17T15:48:02.935 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36413]: finishing. 2024-09-17T15:48:02.935 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36413]: shutting down. 2024-09-17T15:48:02.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.889+0000 7feb47d17640 1 -- 172.21.15.64:0/1217797848 >> v1:172.21.15.100:6812/4210963484 conn(0x7feb24040de0 legacy=0x7feb240432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.889+0000 7feb47d17640 10 monclient: shutdown 2024-09-17T15:48:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.889+0000 7feb47d17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.889+0000 7feb47d17640 1 -- 172.21.15.64:0/1217797848 >> v1:172.21.15.64:6792/0 conn(0x7feb40135c80 legacy=0x7feb40132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.889+0000 7feb462b4640 1 -- 172.21.15.64:0/1217797848 reap_dead start 2024-09-17T15:48:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.889+0000 7feb462b4640 1 -- 172.21.15.64:0/1217797848 reap_dead start 2024-09-17T15:48:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.889+0000 7feb47d17640 1 -- 172.21.15.64:0/1217797848 shutdown_connections 2024-09-17T15:48:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.889+0000 7feb47d17640 1 -- 172.21.15.64:0/1217797848 >> 172.21.15.64:0/1217797848 conn(0x7feb4006daa0 msgr2=0x7feb400a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.889+0000 7feb47d17640 1 -- 172.21.15.64:0/1217797848 shutdown_connections 2024-09-17T15:48:02.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:02.889+0000 7feb47d17640 1 -- 172.21.15.64:0/1217797848 wait complete. 2024-09-17T15:48:02.936 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:02.937 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-17T15:48:02.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-17T15:48:03.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.035+0000 7f4b9700c640 10 monclient: get_monmap_and_config 2024-09-17T15:48:03.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.035+0000 7f4b9700c640 10 monclient: build_initial_monmap 2024-09-17T15:48:03.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.035+0000 7f4b9700c640 10 monclient: monmap: 2024-09-17T15:48:03.038 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:03.038 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.038 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:03.037601+0000 2024-09-17T15:48:03.038 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:03.037601+0000 2024-09-17T15:48:03.039 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:03.039 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.039 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:03.039 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:03.039 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:03.039 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:03.039 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:03.039 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:03.039 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:03.039 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:03.039 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:03.039 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.036+0000 7f4b9700c640 1 Processor -- start 2024-09-17T15:48:03.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.036+0000 7f4b9700c640 1 -- start start 2024-09-17T15:48:03.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.036+0000 7f4b9700c640 10 monclient: init 2024-09-17T15:48:03.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.036+0000 7f4b9700c640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:03.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.036+0000 7f4b9700c640 10 monclient: _add_conns ranks=[4,8,1,7,6,0,2,3,5] 2024-09-17T15:48:03.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b9700c640 10 monclient(hunting): picked mon.noname-c con 0x7f4b900a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:03.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b9700c640 10 monclient(hunting): picked mon.noname-e con 0x7f4b90135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:03.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b9700c640 10 monclient(hunting): picked mon.noname-f con 0x7f4b90135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:03.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b9700c640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9009d6f0 con 0x7f4b90135f70 2024-09-17T15:48:03.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b9700c640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9009d9d0 con 0x7f4b900a5520 2024-09-17T15:48:03.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b9700c640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9009dcb0 con 0x7f4b90135b50 2024-09-17T15:48:03.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b9700c640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:03.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b9700c640 10 monclient(hunting): authenticate will time out at 880.713120s 2024-09-17T15:48:03.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b94da8640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f4b900a5520 0x7f4b900a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:50300/0 (socket says 172.21.15.64:50300) 2024-09-17T15:48:03.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b94da8640 1 -- 172.21.15.64:0/2785332308 learned_addr learned my addr 172.21.15.64:0/2785332308 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:03.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8ffff640 1 --1- 172.21.15.64:0/2785332308 >> v1:172.21.15.64:6793/0 conn(0x7f4b90135b50 0x7f4b9013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:34930/0 (socket says 172.21.15.64:34930) 2024-09-17T15:48:03.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 -- 172.21.15.64:0/2785332308 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2443836669 0 0) 0x7f4b9009d9d0 con 0x7f4b900a5520 2024-09-17T15:48:03.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient(hunting): my global_id is 5098 2024-09-17T15:48:03.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 -- 172.21.15.64:0/2785332308 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b80003520 con 0x7f4b900a5520 2024-09-17T15:48:03.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 -- 172.21.15.64:0/2785332308 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2130709282 0 0) 0x7f4b9009dcb0 con 0x7f4b90135b50 2024-09-17T15:48:03.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient(hunting): my global_id is 5012 2024-09-17T15:48:03.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 -- 172.21.15.64:0/2785332308 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b9009d9d0 con 0x7f4b90135b50 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 -- 172.21.15.64:0/2785332308 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3305118064 0 0) 0x7f4b9009d6f0 con 0x7f4b90135f70 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient(hunting): my global_id is 5149 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 -- 172.21.15.64:0/2785332308 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b9009dcb0 con 0x7f4b90135f70 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 -- 172.21.15.64:0/2785332308 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 957780114 0 0) 0x7f4b80003520 con 0x7f4b900a5520 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 -- 172.21.15.64:0/2785332308 >> v1:172.21.15.64:6793/0 conn(0x7f4b90135b50 legacy=0x7f4b9013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 -- 172.21.15.64:0/2785332308 >> v1:172.21.15.100:6789/0 conn(0x7f4b90135f70 legacy=0x7f4b901415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 monclient: found mon.noname-c 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:48:03.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 -- 172.21.15.64:0/2785332308 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9009ccd0 con 0x7f4b900a5520 2024-09-17T15:48:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:48:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:48:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 -- 172.21.15.64:0/2785332308 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4b84001760 con 0x7f4b900a5520 2024-09-17T15:48:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient: dump: 2024-09-17T15:48:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.051 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b9700c640 5 monclient: authenticate success, global_id 5098 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b9700c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 -- 172.21.15.64:0/2785332308 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b84001900 con 0x7f4b900a5520 2024-09-17T15:48:03.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:03.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b9700c640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:03.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 1 -- 172.21.15.64:0/2785332308 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4b84004be0 con 0x7f4b900a5520 2024-09-17T15:48:03.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b9700c640 4 set_mon_vals no callback set 2024-09-17T15:48:03.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:03.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.037+0000 7f4b8f7fe640 10 monclient: dump: 2024-09-17T15:48:03.053 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.053 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.053 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.053 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.053 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.053 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.053 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.054 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.038+0000 7f4b9700c640 10 monclient: shutdown 2024-09-17T15:48:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.038+0000 7f4b9700c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.038+0000 7f4b9700c640 1 -- 172.21.15.64:0/2785332308 >> v1:172.21.15.64:6791/0 conn(0x7f4b900a5520 legacy=0x7f4b900a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.038+0000 7f4b9700c640 1 -- 172.21.15.64:0/2785332308 shutdown_connections 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.038+0000 7f4b9700c640 1 -- 172.21.15.64:0/2785332308 >> 172.21.15.64:0/2785332308 conn(0x7f4b9006db30 msgr2=0x7f4b900ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.038+0000 7f4b9700c640 1 -- 172.21.15.64:0/2785332308 shutdown_connections 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.038+0000 7f4b9700c640 1 -- 172.21.15.64:0/2785332308 wait complete. 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.039+0000 7f4b9700c640 10 monclient: build_initial_monmap 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.039+0000 7f4b9700c640 10 monclient: monmap: 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:03.055 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:03.056 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:03.056 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:03.056 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:03.056 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:03.056 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:03.056 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:03.056 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.039+0000 7f4b9700c640 1 Processor -- start 2024-09-17T15:48:03.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.039+0000 7f4b9700c640 1 -- start start 2024-09-17T15:48:03.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.039+0000 7f4b9700c640 10 monclient: init 2024-09-17T15:48:03.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.039+0000 7f4b9700c640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:03.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.039+0000 7f4b9700c640 10 monclient: _add_conns ranks=[7,3,4,5,0,6,1,2,8] 2024-09-17T15:48:03.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.039+0000 7f4b9700c640 10 monclient(hunting): picked mon.noname-i con 0x7f4b900a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:03.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 10 monclient(hunting): picked mon.noname-g con 0x7f4b90135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:03.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 10 monclient(hunting): picked mon.noname-c con 0x7f4b90135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:03.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9013ae20 con 0x7f4b90135b50 2024-09-17T15:48:03.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9013a240 con 0x7f4b90135f70 2024-09-17T15:48:03.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b90138ac0 con 0x7f4b900a5520 2024-09-17T15:48:03.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:03.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 10 monclient(hunting): authenticate will time out at 880.716070s 2024-09-17T15:48:03.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b955a9640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f4b90135f70 0x7f4b90137dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:50302/0 (socket says 172.21.15.64:50302) 2024-09-17T15:48:03.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b955a9640 1 -- 172.21.15.64:0/879866005 learned_addr learned my addr 172.21.15.64:0/879866005 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:03.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2980448674 0 0) 0x7f4b9013a240 con 0x7f4b90135f70 2024-09-17T15:48:03.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): my global_id is 5107 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b74003550 con 0x7f4b90135f70 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3687549123 0 0) 0x7f4b90138ac0 con 0x7f4b900a5520 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): my global_id is 4993 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b9013a240 con 0x7f4b900a5520 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4193079623 0 0) 0x7f4b9013ae20 con 0x7f4b90135b50 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): my global_id is 114132 2024-09-17T15:48:03.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b90138ac0 con 0x7f4b90135b50 2024-09-17T15:48:03.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2472630646 0 0) 0x7f4b74003550 con 0x7f4b90135f70 2024-09-17T15:48:03.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b9013ae20 con 0x7f4b90135f70 2024-09-17T15:48:03.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4b88001e10 con 0x7f4b90135f70 2024-09-17T15:48:03.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:03.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:03.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): dump: 2024-09-17T15:48:03.059 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.059 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.059 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.059 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.059 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.059 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2884181683 0 0) 0x7f4b9013a240 con 0x7f4b900a5520 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b74003550 con 0x7f4b900a5520 2024-09-17T15:48:03.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1072187159 0 0) 0x7f4b9013ae20 con 0x7f4b90135f70 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 >> v1:172.21.15.100:6792/0 conn(0x7f4b900a5520 legacy=0x7f4b90136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 >> v1:172.21.15.100:6790/0 conn(0x7f4b90135b50 legacy=0x7f4b901376c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient: _finish_hunting 0 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 monclient: found mon.c 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9013a580 con 0x7f4b90135f70 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 5 monclient: authenticate success, global_id 5107 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 10 monclient: _renew_subs 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4b9013a860 con 0x7f4b90135f70 2024-09-17T15:48:03.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 10 monclient: _renew_subs 2024-09-17T15:48:03.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:03.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b9700c640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f4b901d7160 con 0x7f4b90135f70 2024-09-17T15:48:03.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b880020b0 con 0x7f4b90135f70 2024-09-17T15:48:03.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.040+0000 7f4b8d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:03.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.041+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4b88003f80 con 0x7f4b90135f70 2024-09-17T15:48:03.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.041+0000 7f4b8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.041+0000 7f4b8effd640 4 set_mon_vals no callback set 2024-09-17T15:48:03.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.041+0000 7f4b8d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:03.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.041+0000 7f4b8d7fa640 10 monclient: dump: 2024-09-17T15:48:03.062 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.062 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.062 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.041+0000 7f4b9700c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:03.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.041+0000 7f4b9700c640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4b700065c0 con 0x7f4b70002860 2024-09-17T15:48:03.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.041+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f4b880041a0 con 0x7f4b90135f70 2024-09-17T15:48:03.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(109..109 src has 1..109) ==== 3898+0+0 (unknown 1074856076 0 0) 0x7f4b8804f040 con 0x7f4b90135f70 2024-09-17T15:48:03.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2324535057 0 0) 0x7f4b700065c0 con 0x7f4b70002860 2024-09-17T15:48:03.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 10 monclient(hunting): my global_id is 5158 2024-09-17T15:48:03.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b9013ae20 con 0x7f4b70002860 2024-09-17T15:48:03.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8ffff640 10 monclient: get_auth_request con 0x7f4b74040430 auth_method 0 2024-09-17T15:48:03.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 753384458 0 0) 0x7f4b9013ae20 con 0x7f4b70002860 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b700065c0 con 0x7f4b70002860 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4b84004bc0 con 0x7f4b70002860 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 10 monclient: dump: 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.065 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1253857697 0 0) 0x7f4b700065c0 con 0x7f4b70002860 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.042+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.100:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b7000a160 con 0x7f4b70002860 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.043+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4b84007480 con 0x7f4b70002860 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.043+0000 7f4b8d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.043+0000 7f4b8d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.043+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 >> v1:172.21.15.100:6789/0 conn(0x7f4b70002860 legacy=0x7f4b70004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.043+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 >> v1:172.21.15.100:6789/0 conn(0x7f4b70002860 legacy=0x7f4b70004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:03.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b9700c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:03.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b9700c640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4b70009e70 con 0x7f4b70009130 2024-09-17T15:48:03.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3418549108 0 0) 0x7f4b70009e70 con 0x7f4b70009130 2024-09-17T15:48:03.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b8d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b8d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b8d7fa640 10 monclient(hunting): my global_id is 5167 2024-09-17T15:48:03.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b700065c0 con 0x7f4b70009130 2024-09-17T15:48:03.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1252051664 0 0) 0x7f4b700065c0 con 0x7f4b70009130 2024-09-17T15:48:03.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b70009e70 con 0x7f4b70009130 2024-09-17T15:48:03.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4b84005680 con 0x7f4b70009130 2024-09-17T15:48:03.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b8d7fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:48:03.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.049+0000 7f4b8d7fa640 10 monclient: dump: 2024-09-17T15:48:03.069 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.069 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.069 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.069 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.069 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2276334464 0 0) 0x7f4b70009e70 con 0x7f4b70009130 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 --> v1:172.21.15.100:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4b7000c410 con 0x7f4b70009130 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1739069976) 0x7f4b84006af0 con 0x7f4b70009130 2024-09-17T15:48:03.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b8d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:03.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b8d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:03.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 >> v1:172.21.15.100:6789/0 conn(0x7f4b70009130 legacy=0x7f4b700095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b8d7fa640 1 -- 172.21.15.64:0/879866005 >> v1:172.21.15.100:6789/0 conn(0x7f4b70009130 legacy=0x7f4b700095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:03.071 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:03.071 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 139701, 2024-09-17T15:48:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.072 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:03.073 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:03.074 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:03.075 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:03.076 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.077 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:03.078 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.079 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.080 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:03.081 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:03.082 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.084 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:03.088 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:03.090 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:03.091 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:48:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:03.092 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:48:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.093 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:03.093 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:03.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b9700c640 1 -- 172.21.15.64:0/879866005 >> v1:172.21.15.100:6812/4210963484 conn(0x7f4b74040430 legacy=0x7f4b740428f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b9700c640 10 monclient: shutdown 2024-09-17T15:48:03.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b9700c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:03.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b9700c640 1 -- 172.21.15.64:0/879866005 >> v1:172.21.15.64:6791/0 conn(0x7f4b90135f70 legacy=0x7f4b90137dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b955a9640 1 -- 172.21.15.64:0/879866005 reap_dead start 2024-09-17T15:48:03.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b955a9640 1 -- 172.21.15.64:0/879866005 reap_dead start 2024-09-17T15:48:03.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b9700c640 1 -- 172.21.15.64:0/879866005 shutdown_connections 2024-09-17T15:48:03.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b9700c640 1 -- 172.21.15.64:0/879866005 >> 172.21.15.64:0/879866005 conn(0x7f4b9006db30 msgr2=0x7f4b90133e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:03.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b9700c640 1 -- 172.21.15.64:0/879866005 shutdown_connections 2024-09-17T15:48:03.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.050+0000 7f4b9700c640 1 -- 172.21.15.64:0/879866005 wait complete. 2024-09-17T15:48:03.095 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:03.095 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-17T15:48:03.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-17T15:48:03.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.195+0000 7fac5356b640 10 monclient: get_monmap_and_config 2024-09-17T15:48:03.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.195+0000 7fac5356b640 10 monclient: build_initial_monmap 2024-09-17T15:48:03.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.195+0000 7fac5356b640 10 monclient: monmap: 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:03.196927+0000 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:03.196927+0000 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:03.198 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:03.199 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:03.199 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.195+0000 7fac5356b640 1 Processor -- start 2024-09-17T15:48:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.195+0000 7fac5356b640 1 -- start start 2024-09-17T15:48:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.195+0000 7fac5356b640 10 monclient: init 2024-09-17T15:48:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac5356b640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac5356b640 10 monclient: _add_conns ranks=[3,1,4,8,6,0,5,7,2] 2024-09-17T15:48:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac5356b640 10 monclient(hunting): picked mon.noname-g con 0x7fac4c0a54b0 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac5356b640 10 monclient(hunting): picked mon.noname-f con 0x7fac4c135c80 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac5356b640 10 monclient(hunting): picked mon.noname-c con 0x7fac4c1360a0 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac5356b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac4c0aacb0 con 0x7fac4c135c80 2024-09-17T15:48:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac5356b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac4c09d780 con 0x7fac4c0a54b0 2024-09-17T15:48:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac5356b640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac4c09da60 con 0x7fac4c1360a0 2024-09-17T15:48:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac5356b640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:03.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac5356b640 10 monclient(hunting): authenticate will time out at 880.872412s 2024-09-17T15:48:03.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac51b08640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fac4c1360a0 0x7fac4c141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:50310/0 (socket says 172.21.15.64:50310) 2024-09-17T15:48:03.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac51b08640 1 -- 172.21.15.64:0/3142425448 learned_addr learned my addr 172.21.15.64:0/3142425448 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:03.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 1 -- 172.21.15.64:0/3142425448 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1685850654 0 0) 0x7fac4c09da60 con 0x7fac4c1360a0 2024-09-17T15:48:03.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 10 monclient(hunting): my global_id is 5116 2024-09-17T15:48:03.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 1 -- 172.21.15.64:0/3142425448 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac38003520 con 0x7fac4c1360a0 2024-09-17T15:48:03.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 1 -- 172.21.15.64:0/3142425448 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1147199558 0 0) 0x7fac4c09d780 con 0x7fac4c0a54b0 2024-09-17T15:48:03.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 10 monclient(hunting): my global_id is 114141 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 1 -- 172.21.15.64:0/3142425448 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac4c09da60 con 0x7fac4c0a54b0 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 1 -- 172.21.15.64:0/3142425448 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 33290184 0 0) 0x7fac4c0aacb0 con 0x7fac4c135c80 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 10 monclient(hunting): my global_id is 5176 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.196+0000 7fac43fff640 1 -- 172.21.15.64:0/3142425448 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac4c09d780 con 0x7fac4c135c80 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 1 -- 172.21.15.64:0/3142425448 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3014002883 0 0) 0x7fac38003520 con 0x7fac4c1360a0 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 1 -- 172.21.15.64:0/3142425448 >> v1:172.21.15.100:6790/0 conn(0x7fac4c0a54b0 legacy=0x7fac4c0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 1 -- 172.21.15.64:0/3142425448 >> v1:172.21.15.100:6789/0 conn(0x7fac4c135c80 legacy=0x7fac4c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 10 monclient: _finish_hunting 0 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 1 monclient: found mon.noname-c 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:03.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 1 -- 172.21.15.64:0/3142425448 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c09dd40 con 0x7fac4c1360a0 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 10 monclient: _finish_auth 0 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 10 monclient: _check_auth_tickets 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 1 -- 172.21.15.64:0/3142425448 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fac44001b30 con 0x7fac4c1360a0 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 10 monclient: dump: 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.210 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac5356b640 5 monclient: authenticate success, global_id 5116 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac5356b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 1 -- 172.21.15.64:0/3142425448 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fac44001cd0 con 0x7fac4c1360a0 2024-09-17T15:48:03.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac5356b640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 1 -- 172.21.15.64:0/3142425448 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fac44004f60 con 0x7fac4c1360a0 2024-09-17T15:48:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac5356b640 4 set_mon_vals no callback set 2024-09-17T15:48:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac43fff640 10 monclient: dump: 2024-09-17T15:48:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.212 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.213 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac5356b640 10 monclient: shutdown 2024-09-17T15:48:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac5356b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac5356b640 1 -- 172.21.15.64:0/3142425448 >> v1:172.21.15.64:6791/0 conn(0x7fac4c1360a0 legacy=0x7fac4c141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac5356b640 1 -- 172.21.15.64:0/3142425448 shutdown_connections 2024-09-17T15:48:03.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac5356b640 1 -- 172.21.15.64:0/3142425448 >> 172.21.15.64:0/3142425448 conn(0x7fac4c06daa0 msgr2=0x7fac4c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:03.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac5356b640 1 -- 172.21.15.64:0/3142425448 shutdown_connections 2024-09-17T15:48:03.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.197+0000 7fac5356b640 1 -- 172.21.15.64:0/3142425448 wait complete. 2024-09-17T15:48:03.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.198+0000 7fac5356b640 10 monclient: build_initial_monmap 2024-09-17T15:48:03.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.198+0000 7fac5356b640 10 monclient: monmap: 2024-09-17T15:48:03.214 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:03.214 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:03.214 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:03.214 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:03.214 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:03.214 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:03.214 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:03.214 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:03.215 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:03.215 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:03.215 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:03.215 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:03.215 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:03.215 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:03.215 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:03.215 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.198+0000 7fac5356b640 1 Processor -- start 2024-09-17T15:48:03.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.198+0000 7fac5356b640 1 -- start start 2024-09-17T15:48:03.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.198+0000 7fac5356b640 10 monclient: init 2024-09-17T15:48:03.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac5356b640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:03.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac5356b640 10 monclient: _add_conns ranks=[1,8,4,0,2,6,3,7,5] 2024-09-17T15:48:03.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac5356b640 10 monclient(hunting): picked mon.noname-f con 0x7fac4c0a54b0 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:03.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac5356b640 10 monclient(hunting): picked mon.noname-e con 0x7fac4c135c80 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:03.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac5356b640 10 monclient(hunting): picked mon.noname-c con 0x7fac4c1360a0 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:03.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac5356b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac4c0a05d0 con 0x7fac4c0a54b0 2024-09-17T15:48:03.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac5356b640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac4c09f120 con 0x7fac4c1360a0 2024-09-17T15:48:03.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac5356b640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac4c0a7260 con 0x7fac4c135c80 2024-09-17T15:48:03.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac5356b640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:03.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac5356b640 10 monclient(hunting): authenticate will time out at 880.875308s 2024-09-17T15:48:03.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac50b06640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fac4c135c80 0x7fac4c09ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:34946/0 (socket says 172.21.15.64:34946) 2024-09-17T15:48:03.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac50b06640 1 -- 172.21.15.64:0/3909868454 learned_addr learned my addr 172.21.15.64:0/3909868454 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:03.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac51b08640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fac4c1360a0 0x7fac4c1d62d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:50322/0 (socket says 172.21.15.64:50322) 2024-09-17T15:48:03.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2488132912 0 0) 0x7fac4c09f120 con 0x7fac4c1360a0 2024-09-17T15:48:03.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): my global_id is 5125 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac30003550 con 0x7fac4c1360a0 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1279211590 0 0) 0x7fac4c0a7260 con 0x7fac4c135c80 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): my global_id is 5021 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac4c09f120 con 0x7fac4c135c80 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3755311691 0 0) 0x7fac4c09da60 con 0x7fac4c0a54b0 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): my global_id is 5185 2024-09-17T15:48:03.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac4c0a7260 con 0x7fac4c0a54b0 2024-09-17T15:48:03.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 726726144 0 0) 0x7fac30003550 con 0x7fac4c1360a0 2024-09-17T15:48:03.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac4c09da60 con 0x7fac4c1360a0 2024-09-17T15:48:03.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fac44001a70 con 0x7fac4c1360a0 2024-09-17T15:48:03.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:03.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:03.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): dump: 2024-09-17T15:48:03.218 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.218 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.218 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.218 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.218 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1759371656 0 0) 0x7fac4c09f120 con 0x7fac4c135c80 2024-09-17T15:48:03.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac30003550 con 0x7fac4c135c80 2024-09-17T15:48:03.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fac3c0019c0 con 0x7fac4c135c80 2024-09-17T15:48:03.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:03.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:03.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 10 monclient(hunting): dump: 2024-09-17T15:48:03.220 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.220 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.220 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.220 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.220 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.220 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.220 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.220 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.220 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.221 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.221 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.221 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.221 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.221 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.221 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.221 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 738536247 0 0) 0x7fac4c0a7260 con 0x7fac4c0a54b0 2024-09-17T15:48:03.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac4c09f120 con 0x7fac4c0a54b0 2024-09-17T15:48:03.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.199+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fac48001ad0 con 0x7fac4c0a54b0 2024-09-17T15:48:03.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:03.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:48:03.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 10 monclient(hunting): dump: 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.222 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1808079803 0 0) 0x7fac4c09da60 con 0x7fac4c1360a0 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 >> v1:172.21.15.64:6793/0 conn(0x7fac4c135c80 legacy=0x7fac4c09ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 >> v1:172.21.15.100:6789/0 conn(0x7fac4c0a54b0 legacy=0x7fac4c09e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 1 monclient: found mon.c 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c1353b0 con 0x7fac4c1360a0 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 10 monclient: _finish_auth 0 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:03.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac5356b640 5 monclient: authenticate success, global_id 5125 2024-09-17T15:48:03.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac5356b640 10 monclient: _renew_subs 2024-09-17T15:48:03.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac5356b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:03.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac5356b640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fac4c132980 con 0x7fac4c1360a0 2024-09-17T15:48:03.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fac44004e10 con 0x7fac4c1360a0 2024-09-17T15:48:03.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:03.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fac44005300 con 0x7fac4c1360a0 2024-09-17T15:48:03.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:03.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac41ffb640 10 monclient: dump: 2024-09-17T15:48:03.224 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.224 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.225 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac5356b640 10 monclient: _renew_subs 2024-09-17T15:48:03.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac437fe640 4 set_mon_vals no callback set 2024-09-17T15:48:03.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac5356b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:03.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.200+0000 7fac5356b640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fac4c132300 con 0x7fac4c1360a0 2024-09-17T15:48:03.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fac44011be0 con 0x7fac4c1360a0 2024-09-17T15:48:03.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac5356b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:03.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac5356b640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fac4c134300 con 0x7fac4c09f670 2024-09-17T15:48:03.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(109..109 src has 1..109) ==== 3898+0+0 (unknown 1074856076 0 0) 0x7fac4404eda0 con 0x7fac4c1360a0 2024-09-17T15:48:03.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac50b06640 10 monclient: get_auth_request con 0x7fac30040e10 auth_method 0 2024-09-17T15:48:03.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2720353385 0 0) 0x7fac4c09f120 con 0x7fac4c09f670 2024-09-17T15:48:03.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 10 monclient(hunting): my global_id is 114150 2024-09-17T15:48:03.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac4c09da60 con 0x7fac4c09f670 2024-09-17T15:48:03.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3795804254 0 0) 0x7fac4c09da60 con 0x7fac4c09f670 2024-09-17T15:48:03.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac4c09f120 con 0x7fac4c09f670 2024-09-17T15:48:03.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fac480039b0 con 0x7fac4c09f670 2024-09-17T15:48:03.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:03.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 10 monclient: dump: 2024-09-17T15:48:03.227 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.227 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.227 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.227 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.228 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.228 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.228 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.228 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.228 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.228 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.228 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.228 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.228 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.228 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.228 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.228 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1213864501 0 0) 0x7fac4c09f120 con 0x7fac4c09f670 2024-09-17T15:48:03.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.201+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.100:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac4c077070 con 0x7fac4c09f670 2024-09-17T15:48:03.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.202+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fac48003d50 con 0x7fac4c09f670 2024-09-17T15:48:03.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.202+0000 7fac41ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:03.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.202+0000 7fac41ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:03.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.202+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 >> v1:172.21.15.100:6790/0 conn(0x7fac4c09f670 legacy=0x7fac4c12d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.202+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 >> v1:172.21.15.100:6790/0 conn(0x7fac4c09f670 legacy=0x7fac4c12d6a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.208+0000 7fac5356b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.208+0000 7fac5356b640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fac4c0abea0 con 0x7fac4c02e450 2024-09-17T15:48:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.208+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 369450648 0 0) 0x7fac48003f90 con 0x7fac4c02e450 2024-09-17T15:48:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.208+0000 7fac41ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.208+0000 7fac41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.208+0000 7fac41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.208+0000 7fac41ffb640 10 monclient(hunting): my global_id is 114159 2024-09-17T15:48:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.208+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac4c09f120 con 0x7fac4c02e450 2024-09-17T15:48:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4057641843 0 0) 0x7fac4c09f120 con 0x7fac4c02e450 2024-09-17T15:48:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac48003f90 con 0x7fac4c02e450 2024-09-17T15:48:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fac48004750 con 0x7fac4c02e450 2024-09-17T15:48:03.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac41ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac41ffb640 10 monclient: dump: 2024-09-17T15:48:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.232 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.233 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 649840611 0 0) 0x7fac48003f90 con 0x7fac4c02e450 2024-09-17T15:48:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 --> v1:172.21.15.100:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac4c142920 con 0x7fac4c02e450 2024-09-17T15:48:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 2316522750) 0x7fac480049b0 con 0x7fac4c02e450 2024-09-17T15:48:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac41ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac41ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 >> v1:172.21.15.100:6790/0 conn(0x7fac4c02e450 legacy=0x7fac4c12ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac41ffb640 1 -- 172.21.15.64:0/3909868454 >> v1:172.21.15.100:6790/0 conn(0x7fac4c02e450 legacy=0x7fac4c12ddb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:03.234 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 6432, 2024-09-17T15:48:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:03.234 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:03.235 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:03.236 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:03.237 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:03.238 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.239 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.240 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.243 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.244 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.245 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:03.246 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.248 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:03.250 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.252 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.253 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:03.254 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:03.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac5356b640 1 -- 172.21.15.64:0/3909868454 >> v1:172.21.15.100:6812/4210963484 conn(0x7fac30040e10 legacy=0x7fac300432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac5356b640 10 monclient: shutdown 2024-09-17T15:48:03.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac5356b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:03.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac5356b640 1 -- 172.21.15.64:0/3909868454 >> v1:172.21.15.64:6791/0 conn(0x7fac4c1360a0 legacy=0x7fac4c1d62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac51b08640 1 -- 172.21.15.64:0/3909868454 reap_dead start 2024-09-17T15:48:03.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac51b08640 1 -- 172.21.15.64:0/3909868454 reap_dead start 2024-09-17T15:48:03.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac5356b640 1 -- 172.21.15.64:0/3909868454 shutdown_connections 2024-09-17T15:48:03.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.209+0000 7fac5356b640 1 -- 172.21.15.64:0/3909868454 >> 172.21.15.64:0/3909868454 conn(0x7fac4c06daa0 msgr2=0x7fac4c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:03.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.210+0000 7fac5356b640 1 -- 172.21.15.64:0/3909868454 shutdown_connections 2024-09-17T15:48:03.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.210+0000 7fac5356b640 1 -- 172.21.15.64:0/3909868454 wait complete. 2024-09-17T15:48:03.256 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:03.256 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-17T15:48:03.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-17T15:48:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.356+0000 7f0f2143c640 10 monclient: get_monmap_and_config 2024-09-17T15:48:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.356+0000 7f0f2143c640 10 monclient: build_initial_monmap 2024-09-17T15:48:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.356+0000 7f0f2143c640 10 monclient: monmap: 2024-09-17T15:48:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:03.358045+0000 2024-09-17T15:48:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:03.358045+0000 2024-09-17T15:48:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:03.361 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:03.361 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:03.361 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:03.361 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.356+0000 7f0f2143c640 1 Processor -- start 2024-09-17T15:48:03.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.356+0000 7f0f2143c640 1 -- start start 2024-09-17T15:48:03.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.356+0000 7f0f2143c640 10 monclient: init 2024-09-17T15:48:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f2143c640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f2143c640 10 monclient: _add_conns ranks=[5,7,0,2,1,6,8,4,3] 2024-09-17T15:48:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f2143c640 10 monclient(hunting): picked mon.noname-h con 0x7f0f1c0a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:03.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f2143c640 10 monclient(hunting): picked mon.noname-i con 0x7f0f1c135c80 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f2143c640 10 monclient(hunting): picked mon.noname-a con 0x7f0f1c1360a0 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f2143c640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f1c0aacb0 con 0x7f0f1c1360a0 2024-09-17T15:48:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f2143c640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f1c09d780 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f2143c640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f1c09da60 con 0x7f0f1c135c80 2024-09-17T15:48:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f2143c640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f2143c640 10 monclient(hunting): authenticate will time out at 881.033380s 2024-09-17T15:48:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f1b7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f0f1c1360a0 0x7f0f1c141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:47024/0 (socket says 172.21.15.64:47024) 2024-09-17T15:48:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f1b7fe640 1 -- 172.21.15.64:0/664181525 learned_addr learned my addr 172.21.15.64:0/664181525 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 1 -- 172.21.15.64:0/664181525 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3148292983 0 0) 0x7f0f1c0aacb0 con 0x7f0f1c1360a0 2024-09-17T15:48:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient(hunting): my global_id is 5067 2024-09-17T15:48:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 1 -- 172.21.15.64:0/664181525 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f08003520 con 0x7f0f1c1360a0 2024-09-17T15:48:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 1 -- 172.21.15.64:0/664181525 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2555646489 0 0) 0x7f0f1c09da60 con 0x7f0f1c135c80 2024-09-17T15:48:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient(hunting): my global_id is 5002 2024-09-17T15:48:03.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 1 -- 172.21.15.64:0/664181525 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f1c0aacb0 con 0x7f0f1c135c80 2024-09-17T15:48:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 1 -- 172.21.15.64:0/664181525 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2307866210 0 0) 0x7f0f1c09d780 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient(hunting): my global_id is 114161 2024-09-17T15:48:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 1 -- 172.21.15.64:0/664181525 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f1c09da60 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 1 -- 172.21.15.64:0/664181525 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 925134094 0 0) 0x7f0f08003520 con 0x7f0f1c1360a0 2024-09-17T15:48:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 1 -- 172.21.15.64:0/664181525 >> v1:172.21.15.100:6792/0 conn(0x7f0f1c135c80 legacy=0x7f0f1c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 1 -- 172.21.15.64:0/664181525 >> v1:172.21.15.100:6791/0 conn(0x7f0f1c0a54b0 legacy=0x7f0f1c0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:48:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 1 monclient: found mon.noname-a 2024-09-17T15:48:03.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:48:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 1 -- 172.21.15.64:0/664181525 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1c09dd40 con 0x7f0f1c1360a0 2024-09-17T15:48:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient: _finish_auth 0 2024-09-17T15:48:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:48:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.357+0000 7f0f19ffb640 1 -- 172.21.15.64:0/664181525 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0f0c001bb0 con 0x7f0f1c1360a0 2024-09-17T15:48:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f19ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f19ffb640 10 monclient: dump: 2024-09-17T15:48:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.374 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.375 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f2143c640 5 monclient: authenticate success, global_id 5067 2024-09-17T15:48:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f2143c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f19ffb640 1 -- 172.21.15.64:0/664181525 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0f0c001dd0 con 0x7f0f1c1360a0 2024-09-17T15:48:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f19ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f19ffb640 1 -- 172.21.15.64:0/664181525 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0f0c004f60 con 0x7f0f1c1360a0 2024-09-17T15:48:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f2143c640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f2143c640 4 set_mon_vals no callback set 2024-09-17T15:48:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f19ffb640 10 monclient: dump: 2024-09-17T15:48:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.376 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f2143c640 10 monclient: shutdown 2024-09-17T15:48:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f2143c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f2143c640 1 -- 172.21.15.64:0/664181525 >> v1:172.21.15.64:6789/0 conn(0x7f0f1c1360a0 legacy=0x7f0f1c141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f2143c640 1 -- 172.21.15.64:0/664181525 shutdown_connections 2024-09-17T15:48:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f2143c640 1 -- 172.21.15.64:0/664181525 >> 172.21.15.64:0/664181525 conn(0x7f0f1c06daa0 msgr2=0x7f0f1c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f2143c640 1 -- 172.21.15.64:0/664181525 shutdown_connections 2024-09-17T15:48:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.358+0000 7f0f2143c640 1 -- 172.21.15.64:0/664181525 wait complete. 2024-09-17T15:48:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.359+0000 7f0f2143c640 10 monclient: build_initial_monmap 2024-09-17T15:48:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.359+0000 7f0f2143c640 10 monclient: monmap: 2024-09-17T15:48:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:03.378 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:03.379 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.359+0000 7f0f2143c640 1 Processor -- start 2024-09-17T15:48:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.359+0000 7f0f2143c640 1 -- start start 2024-09-17T15:48:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.359+0000 7f0f2143c640 10 monclient: init 2024-09-17T15:48:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f2143c640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f2143c640 10 monclient: _add_conns ranks=[6,8,7,3,0,4,1,2,5] 2024-09-17T15:48:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f2143c640 10 monclient(hunting): picked mon.noname-d con 0x7f0f1c0a54b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f2143c640 10 monclient(hunting): picked mon.noname-e con 0x7f0f1c135c80 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f2143c640 10 monclient(hunting): picked mon.noname-i con 0x7f0f1c1360a0 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f2143c640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f1c0a0530 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f2143c640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f1c09f120 con 0x7f0f1c1360a0 2024-09-17T15:48:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f2143c640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f1c0a0af0 con 0x7f0f1c135c80 2024-09-17T15:48:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f2143c640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f2143c640 10 monclient(hunting): authenticate will time out at 881.036466s 2024-09-17T15:48:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f1affd640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f0f1c0a54b0 0x7f0f1c09e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:39382/0 (socket says 172.21.15.64:39382) 2024-09-17T15:48:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f1affd640 1 -- 172.21.15.64:0/1369150308 learned_addr learned my addr 172.21.15.64:0/1369150308 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 564441902 0 0) 0x7f0f1c0a0af0 con 0x7f0f1c135c80 2024-09-17T15:48:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 10 monclient(hunting): my global_id is 5030 2024-09-17T15:48:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0efc003520 con 0x7f0f1c135c80 2024-09-17T15:48:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2627115728 0 0) 0x7f0f1c0a0530 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 10 monclient(hunting): my global_id is 114171 2024-09-17T15:48:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f1c0a0af0 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2587180936 0 0) 0x7f0f1c09f120 con 0x7f0f1c1360a0 2024-09-17T15:48:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 10 monclient(hunting): my global_id is 5011 2024-09-17T15:48:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.360+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f1c0a0530 con 0x7f0f1c1360a0 2024-09-17T15:48:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3461674398 0 0) 0x7f0f1c0a0af0 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f1c09f120 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3054847504 0 0) 0x7f0efc003520 con 0x7f0f1c135c80 2024-09-17T15:48:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f1c0a0af0 con 0x7f0f1c135c80 2024-09-17T15:48:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0f14001b90 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient(hunting): dump: 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.383 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.384 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0f04001ad0 con 0x7f0f1c135c80 2024-09-17T15:48:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:03.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient(hunting): dump: 2024-09-17T15:48:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.386 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2431469192 0 0) 0x7f0f1c0a0530 con 0x7f0f1c1360a0 2024-09-17T15:48:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0efc003520 con 0x7f0f1c1360a0 2024-09-17T15:48:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2813093228 0 0) 0x7f0f1c09f120 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 >> v1:172.21.15.64:6793/0 conn(0x7f0f1c135c80 legacy=0x7f0f1c09ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 >> v1:172.21.15.100:6792/0 conn(0x7f0f1c1360a0 legacy=0x7f0f1c1d6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient: _finish_hunting 0 2024-09-17T15:48:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 monclient: found mon.d 2024-09-17T15:48:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:48:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1c0a6f50 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient: _finish_auth 0 2024-09-17T15:48:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient: _check_auth_tickets 2024-09-17T15:48:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f2143c640 5 monclient: authenticate success, global_id 114171 2024-09-17T15:48:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f2143c640 10 monclient: _renew_subs 2024-09-17T15:48:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f2143c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:48:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f2143c640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0f1c0a0250 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f2143c640 10 monclient: _renew_subs 2024-09-17T15:48:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f2143c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:48:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f2143c640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f0f1c1d7750 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0f140038c0 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0f14003e70 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:48:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f197fa640 4 set_mon_vals no callback set 2024-09-17T15:48:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.361+0000 7f0f137fe640 10 monclient: dump: 2024-09-17T15:48:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.390 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.362+0000 7f0f2143c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.362+0000 7f0f2143c640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0f000065c0 con 0x7f0f00002860 2024-09-17T15:48:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.362+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f0f14004010 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.362+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(109..109 src has 1..109) ==== 3898+0+0 (unknown 1074856076 0 0) 0x7f0f1404ec40 con 0x7f0f1c0a54b0 2024-09-17T15:48:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.362+0000 7f0f1b7fe640 10 monclient: get_auth_request con 0x7f0efc040b50 auth_method 0 2024-09-17T15:48:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.362+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048506963 0 0) 0x7f0f000065c0 con 0x7f0f00002860 2024-09-17T15:48:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.362+0000 7f0f137fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.362+0000 7f0f137fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.362+0000 7f0f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.362+0000 7f0f137fe640 10 monclient(hunting): my global_id is 114170 2024-09-17T15:48:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.362+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f1c09f120 con 0x7f0f00002860 2024-09-17T15:48:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.362+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1200490088 0 0) 0x7f0f1c09f120 con 0x7f0f00002860 2024-09-17T15:48:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.363+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f000065c0 con 0x7f0f00002860 2024-09-17T15:48:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.363+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0f04003c80 con 0x7f0f00002860 2024-09-17T15:48:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.363+0000 7f0f137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.363+0000 7f0f137fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:48:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.363+0000 7f0f137fe640 10 monclient: dump: 2024-09-17T15:48:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.392 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.363+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1701119398 0 0) 0x7f0f000065c0 con 0x7f0f00002860 2024-09-17T15:48:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.363+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.100:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f0000a160 con 0x7f0f00002860 2024-09-17T15:48:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.363+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0f040051c0 con 0x7f0f00002860 2024-09-17T15:48:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.363+0000 7f0f137fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.363+0000 7f0f137fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.363+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 >> v1:172.21.15.100:6791/0 conn(0x7f0f00002860 legacy=0x7f0f00004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.363+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 >> v1:172.21.15.100:6791/0 conn(0x7f0f00002860 legacy=0x7f0f00004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.369+0000 7f0f2143c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.369+0000 7f0f2143c640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0f00009e20 con 0x7f0f000090e0 2024-09-17T15:48:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3297964452 0 0) 0x7f0f00009e20 con 0x7f0f000090e0 2024-09-17T15:48:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 10 monclient(hunting): my global_id is 114179 2024-09-17T15:48:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f000065c0 con 0x7f0f000090e0 2024-09-17T15:48:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3215127853 0 0) 0x7f0f000065c0 con 0x7f0f000090e0 2024-09-17T15:48:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f00009e20 con 0x7f0f000090e0 2024-09-17T15:48:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0f04005960 con 0x7f0f000090e0 2024-09-17T15:48:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:48:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 10 monclient: dump: 2024-09-17T15:48:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.397 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3589243238 0 0) 0x7f0f00009e20 con 0x7f0f000090e0 2024-09-17T15:48:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 --> v1:172.21.15.100:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0f0000c3f0 con 0x7f0f000090e0 2024-09-17T15:48:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 231194479) 0x7f0f04005a10 con 0x7f0f000090e0 2024-09-17T15:48:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 >> v1:172.21.15.100:6791/0 conn(0x7f0f000090e0 legacy=0x7f0f00009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.370+0000 7f0f137fe640 1 -- 172.21.15.64:0/1369150308 >> v1:172.21.15.100:6791/0 conn(0x7f0f000090e0 legacy=0x7f0f00009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:03.398 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 6591, 2024-09-17T15:48:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:48:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:03.418 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:03.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.371+0000 7f0f2143c640 1 -- 172.21.15.64:0/1369150308 >> v1:172.21.15.100:6812/4210963484 conn(0x7f0efc040b50 legacy=0x7f0efc043010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.371+0000 7f0f2143c640 10 monclient: shutdown 2024-09-17T15:48:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.371+0000 7f0f2143c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.371+0000 7f0f2143c640 1 -- 172.21.15.64:0/1369150308 >> v1:172.21.15.64:6792/0 conn(0x7f0f1c0a54b0 legacy=0x7f0f1c09e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.371+0000 7f0f1b7fe640 1 -- 172.21.15.64:0/1369150308 reap_dead start 2024-09-17T15:48:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.371+0000 7f0f1b7fe640 1 -- 172.21.15.64:0/1369150308 reap_dead start 2024-09-17T15:48:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.371+0000 7f0f2143c640 1 -- 172.21.15.64:0/1369150308 shutdown_connections 2024-09-17T15:48:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.371+0000 7f0f2143c640 1 -- 172.21.15.64:0/1369150308 >> 172.21.15.64:0/1369150308 conn(0x7f0f1c06daa0 msgr2=0x7f0f1c12c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.371+0000 7f0f2143c640 1 -- 172.21.15.64:0/1369150308 shutdown_connections 2024-09-17T15:48:03.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.371+0000 7f0f2143c640 1 -- 172.21.15.64:0/1369150308 wait complete. 2024-09-17T15:48:03.421 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:03.421 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-17T15:48:03.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-17T15:48:03.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.520+0000 7f705cf9b640 10 monclient: get_monmap_and_config 2024-09-17T15:48:03.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.520+0000 7f705cf9b640 10 monclient: build_initial_monmap 2024-09-17T15:48:03.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.520+0000 7f705cf9b640 10 monclient: monmap: 2024-09-17T15:48:03.523 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:03.523 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.523 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:03.522031+0000 2024-09-17T15:48:03.523 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:03.522031+0000 2024-09-17T15:48:03.523 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:03.523 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.523 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:03.523 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:03.524 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:03.524 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:03.524 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:03.524 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:03.524 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:03.524 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:03.524 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:03.524 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.520+0000 7f705cf9b640 1 Processor -- start 2024-09-17T15:48:03.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.520+0000 7f705cf9b640 1 -- start start 2024-09-17T15:48:03.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.520+0000 7f705cf9b640 10 monclient: init 2024-09-17T15:48:03.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.521+0000 7f705cf9b640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:03.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.521+0000 7f705cf9b640 10 monclient: _add_conns ranks=[5,7,3,8,0,4,1,6,2] 2024-09-17T15:48:03.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.521+0000 7f705cf9b640 10 monclient(hunting): picked mon.noname-h con 0x7f70580a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:03.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.521+0000 7f705cf9b640 10 monclient(hunting): picked mon.noname-i con 0x7f7058135c80 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:03.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.521+0000 7f705cf9b640 10 monclient(hunting): picked mon.noname-g con 0x7f70581360a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:03.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.521+0000 7f705cf9b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70580aacb0 con 0x7f70581360a0 2024-09-17T15:48:03.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.521+0000 7f705cf9b640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f705809d780 con 0x7f70580a54b0 2024-09-17T15:48:03.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.521+0000 7f705cf9b640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f705809da60 con 0x7f7058135c80 2024-09-17T15:48:03.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.521+0000 7f705cf9b640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:03.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.521+0000 7f705cf9b640 10 monclient(hunting): authenticate will time out at 881.197518s 2024-09-17T15:48:03.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.521+0000 7f705659c640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7f70580a54b0 0x7f70580a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:42472/0 (socket says 172.21.15.64:42472) 2024-09-17T15:48:03.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.521+0000 7f705659c640 1 -- 172.21.15.64:0/137213209 learned_addr learned my addr 172.21.15.64:0/137213209 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:03.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.521+0000 7f7056d9d640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f70581360a0 0x7f7058141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:46776/0 (socket says 172.21.15.64:46776) 2024-09-17T15:48:03.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 -- 172.21.15.64:0/137213209 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1339079775 0 0) 0x7f70580aacb0 con 0x7f70581360a0 2024-09-17T15:48:03.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient(hunting): my global_id is 114168 2024-09-17T15:48:03.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 -- 172.21.15.64:0/137213209 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7044003520 con 0x7f70581360a0 2024-09-17T15:48:03.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 -- 172.21.15.64:0/137213209 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3653009682 0 0) 0x7f705809d780 con 0x7f70580a54b0 2024-09-17T15:48:03.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient(hunting): my global_id is 114188 2024-09-17T15:48:03.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 -- 172.21.15.64:0/137213209 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70580aacb0 con 0x7f70580a54b0 2024-09-17T15:48:03.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 -- 172.21.15.64:0/137213209 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1239299033 0 0) 0x7f705809da60 con 0x7f7058135c80 2024-09-17T15:48:03.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient(hunting): my global_id is 5020 2024-09-17T15:48:03.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 -- 172.21.15.64:0/137213209 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f705809d780 con 0x7f7058135c80 2024-09-17T15:48:03.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 -- 172.21.15.64:0/137213209 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3099694157 0 0) 0x7f7044003520 con 0x7f70581360a0 2024-09-17T15:48:03.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 -- 172.21.15.64:0/137213209 >> v1:172.21.15.100:6792/0 conn(0x7f7058135c80 legacy=0x7f705813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 -- 172.21.15.64:0/137213209 >> v1:172.21.15.100:6791/0 conn(0x7f70580a54b0 legacy=0x7f70580a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:48:03.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 monclient: found mon.noname-g 2024-09-17T15:48:03.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.100:6790/0 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 -- 172.21.15.64:0/137213209 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705809dd40 con 0x7f70581360a0 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient: _finish_auth 0 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 -- 172.21.15.64:0/137213209 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7050001bb0 con 0x7f70581360a0 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient: dump: 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.538 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.539 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.539 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.539 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.539 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.539 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.539 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.539 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.539 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.539 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.539 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f705cf9b640 5 monclient: authenticate success, global_id 114168 2024-09-17T15:48:03.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f705cf9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 -- 172.21.15.64:0/137213209 <== mon.3 v1:172.21.15.100:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7050001d50 con 0x7f70581360a0 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f705cf9b640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f705cf9b640 4 set_mon_vals no callback set 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 1 -- 172.21.15.64:0/137213209 <== mon.3 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7050004ea0 con 0x7f70581360a0 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f7055d9b640 10 monclient: dump: 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.540 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.541 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.541 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.541 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.541 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.541 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.541 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.541 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.541 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.541 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.541 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f705cf9b640 10 monclient: shutdown 2024-09-17T15:48:03.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f705cf9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:03.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f705cf9b640 1 -- 172.21.15.64:0/137213209 >> v1:172.21.15.100:6790/0 conn(0x7f70581360a0 legacy=0x7f7058141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f705cf9b640 1 -- 172.21.15.64:0/137213209 shutdown_connections 2024-09-17T15:48:03.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.522+0000 7f705cf9b640 1 -- 172.21.15.64:0/137213209 >> 172.21.15.64:0/137213209 conn(0x7f705806daa0 msgr2=0x7f70580ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:03.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.523+0000 7f705cf9b640 1 -- 172.21.15.64:0/137213209 shutdown_connections 2024-09-17T15:48:03.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.523+0000 7f705cf9b640 1 -- 172.21.15.64:0/137213209 wait complete. 2024-09-17T15:48:03.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.523+0000 7f705cf9b640 10 monclient: build_initial_monmap 2024-09-17T15:48:03.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.523+0000 7f705cf9b640 10 monclient: monmap: 2024-09-17T15:48:03.542 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:03.542 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:03.542 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:03.542 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:03.542 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:03.543 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:03.543 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:03.543 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:03.543 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:03.543 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:03.543 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:03.543 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:03.543 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:03.543 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:03.543 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:03.543 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705cf9b640 1 Processor -- start 2024-09-17T15:48:03.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705cf9b640 1 -- start start 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705cf9b640 10 monclient: init 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705cf9b640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705cf9b640 10 monclient: _add_conns ranks=[8,2,1,3,0,5,6,7,4] 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705cf9b640 10 monclient(hunting): picked mon.noname-e con 0x7f70580a54b0 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705cf9b640 10 monclient(hunting): picked mon.noname-b con 0x7f7058135c80 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705cf9b640 10 monclient(hunting): picked mon.noname-f con 0x7f70581360a0 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705cf9b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70580a5250 con 0x7f70581360a0 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705cf9b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70580a0e20 con 0x7f7058135c80 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705cf9b640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70580a1d80 con 0x7f70580a54b0 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705cf9b640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705cf9b640 10 monclient(hunting): authenticate will time out at 881.200683s 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705659c640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f70580a54b0 0x7f70580a4b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:34950/0 (socket says 172.21.15.64:34950) 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f705659c640 1 -- 172.21.15.64:0/2567126730 learned_addr learned my addr 172.21.15.64:0/2567126730 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:03.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3910224538 0 0) 0x7f70580a0e20 con 0x7f7058135c80 2024-09-17T15:48:03.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f704effd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f704effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f704effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.524+0000 7f704effd640 10 monclient(hunting): my global_id is 4871 2024-09-17T15:48:03.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7038003520 con 0x7f7058135c80 2024-09-17T15:48:03.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3952769151 0 0) 0x7f70580a1d80 con 0x7f70580a54b0 2024-09-17T15:48:03.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): my global_id is 5039 2024-09-17T15:48:03.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70580a0e20 con 0x7f70580a54b0 2024-09-17T15:48:03.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1959569213 0 0) 0x7f70580a5250 con 0x7f70581360a0 2024-09-17T15:48:03.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): my global_id is 5194 2024-09-17T15:48:03.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70580a1d80 con 0x7f70581360a0 2024-09-17T15:48:03.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2961704831 0 0) 0x7f70580a0e20 con 0x7f70580a54b0 2024-09-17T15:48:03.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70580a5250 con 0x7f70580a54b0 2024-09-17T15:48:03.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7048001650 con 0x7f70580a54b0 2024-09-17T15:48:03.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:03.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:48:03.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): dump: 2024-09-17T15:48:03.547 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.547 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.547 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.547 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.547 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.547 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.547 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.547 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.547 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.547 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.547 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.548 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.548 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.548 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.548 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.548 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:03.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3878986982 0 0) 0x7f7038003520 con 0x7f7058135c80 2024-09-17T15:48:03.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70580a0e20 con 0x7f7058135c80 2024-09-17T15:48:03.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2182400707 0 0) 0x7f70580a1d80 con 0x7f70581360a0 2024-09-17T15:48:03.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7038003520 con 0x7f70581360a0 2024-09-17T15:48:03.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f70400013a0 con 0x7f7058135c80 2024-09-17T15:48:03.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:03.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:03.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): dump: 2024-09-17T15:48:03.549 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.549 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.549 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.549 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.549 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.549 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.549 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.549 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.549 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.549 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.549 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7050001b70 con 0x7f70581360a0 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient(hunting): dump: 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.550 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.551 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.551 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.551 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.551 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.551 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.551 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.551 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.551 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.551 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.552 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.552 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1705939927 0 0) 0x7f70580a5250 con 0x7f70580a54b0 2024-09-17T15:48:03.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 >> v1:172.21.15.64:6790/0 conn(0x7f7058135c80 legacy=0x7f70581d2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 >> v1:172.21.15.100:6789/0 conn(0x7f70581360a0 legacy=0x7f70581d5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient: _finish_hunting 0 2024-09-17T15:48:03.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 monclient: found mon.e 2024-09-17T15:48:03.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:03.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:03.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70580a2060 con 0x7f70580a54b0 2024-09-17T15:48:03.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient: _finish_auth 0 2024-09-17T15:48:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f705cf9b640 5 monclient: authenticate success, global_id 5039 2024-09-17T15:48:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f705cf9b640 10 monclient: _renew_subs 2024-09-17T15:48:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f705cf9b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f705cf9b640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70580a2620 con 0x7f70580a54b0 2024-09-17T15:48:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f705cf9b640 10 monclient: _renew_subs 2024-09-17T15:48:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f705cf9b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f705cf9b640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f70580a0890 con 0x7f70580a54b0 2024-09-17T15:48:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7048004a90 con 0x7f70580a54b0 2024-09-17T15:48:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7048004f80 con 0x7f70580a54b0 2024-09-17T15:48:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f705559a640 4 set_mon_vals no callback set 2024-09-17T15:48:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.525+0000 7f704effd640 10 monclient: dump: 2024-09-17T15:48:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.555 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.526+0000 7f705cf9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.526+0000 7f705cf9b640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f703c0065c0 con 0x7f703c002860 2024-09-17T15:48:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.526+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f7048011860 con 0x7f70580a54b0 2024-09-17T15:48:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.526+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(109..109 src has 1..109) ==== 3898+0+0 (unknown 1074856076 0 0) 0x7f704804ec50 con 0x7f70580a54b0 2024-09-17T15:48:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.526+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 828665040 0 0) 0x7f703c0065c0 con 0x7f703c002860 2024-09-17T15:48:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.526+0000 7f704effd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.526+0000 7f704effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.526+0000 7f704effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.526+0000 7f704effd640 10 monclient(hunting): my global_id is 5029 2024-09-17T15:48:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.526+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70580a5250 con 0x7f703c002860 2024-09-17T15:48:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.526+0000 7f7056d9d640 10 monclient: get_auth_request con 0x7f7038040d30 auth_method 0 2024-09-17T15:48:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.527+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1347690239 0 0) 0x7f70580a5250 con 0x7f703c002860 2024-09-17T15:48:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.527+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f703c0065c0 con 0x7f703c002860 2024-09-17T15:48:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.527+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7040003930 con 0x7f703c002860 2024-09-17T15:48:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.527+0000 7f704effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.527+0000 7f704effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:48:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.527+0000 7f704effd640 10 monclient: dump: 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.557 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.558 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.527+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3963300099 0 0) 0x7f703c0065c0 con 0x7f703c002860 2024-09-17T15:48:03.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.527+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.100:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f703c00a160 con 0x7f703c002860 2024-09-17T15:48:03.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.527+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7040005bb0 con 0x7f703c002860 2024-09-17T15:48:03.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.527+0000 7f704effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:03.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.527+0000 7f704effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:03.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.527+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 >> v1:172.21.15.100:6792/0 conn(0x7f703c002860 legacy=0x7f703c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.527+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 >> v1:172.21.15.100:6792/0 conn(0x7f703c002860 legacy=0x7f703c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:03.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.533+0000 7f705cf9b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:03.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.533+0000 7f705cf9b640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f703c009e50 con 0x7f703c009110 2024-09-17T15:48:03.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 853078437 0 0) 0x7f703c009e50 con 0x7f703c009110 2024-09-17T15:48:03.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 10 monclient(hunting): my global_id is 5038 2024-09-17T15:48:03.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f703c0065c0 con 0x7f703c009110 2024-09-17T15:48:03.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 571853443 0 0) 0x7f703c0065c0 con 0x7f703c009110 2024-09-17T15:48:03.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f703c009e50 con 0x7f703c009110 2024-09-17T15:48:03.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7040005460 con 0x7f703c009110 2024-09-17T15:48:03.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:48:03.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 10 monclient: dump: 2024-09-17T15:48:03.561 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.561 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.561 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.561 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.561 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.561 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.561 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.561 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.561 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.561 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.561 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.562 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.562 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.562 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.562 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.562 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 980404392 0 0) 0x7f703c009e50 con 0x7f703c009110 2024-09-17T15:48:03.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.534+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 --> v1:172.21.15.100:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f703c00c3a0 con 0x7f703c009110 2024-09-17T15:48:03.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 779299227) 0x7f7040005590 con 0x7f703c009110 2024-09-17T15:48:03.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f704effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:03.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f704effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:03.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 >> v1:172.21.15.100:6792/0 conn(0x7f703c009110 legacy=0x7f703c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f704effd640 1 -- 172.21.15.64:0/2567126730 >> v1:172.21.15.100:6792/0 conn(0x7f703c009110 legacy=0x7f703c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:03.563 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:03.563 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:03.563 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:03.563 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:03.563 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:03.563 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 140170, 2024-09-17T15:48:03.563 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:03.563 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:03.563 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:03.563 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:03.563 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:03.564 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:03.564 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:03.564 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:03.564 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:03.564 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:03.564 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.564 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:48:03.564 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:03.564 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:03.564 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:03.564 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:03.564 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:03.564 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:03.565 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:03.565 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:03.565 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:03.565 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:03.565 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:03.565 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:03.565 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:03.565 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:03.565 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.565 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:03.565 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:03.565 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:03.565 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:03.566 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:03.566 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:03.566 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:03.566 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:03.566 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:03.566 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:03.566 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:03.566 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:03.566 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:03.566 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.566 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:03.566 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:03.567 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:03.567 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:03.567 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:03.567 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:03.567 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:03.567 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:03.567 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:03.567 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:03.567 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:03.567 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:03.567 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:03.567 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:03.568 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:03.568 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:03.568 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:03.568 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:03.568 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:03.568 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:03.568 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:03.568 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:03.568 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:03.568 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:03.568 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:03.568 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:03.568 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:03.569 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:03.569 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.569 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:03.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.569 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:03.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.569 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:03.569 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:03.569 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:03.570 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.571 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.571 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:03.571 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.571 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.571 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:03.572 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:03.572 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.572 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.572 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.572 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:03.572 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:03.572 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.572 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.572 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.572 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:03.573 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.573 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.573 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:03.573 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:03.573 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.573 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.573 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.573 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:03.573 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.577 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:03.577 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.577 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:03.577 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:03.577 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.577 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.577 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.577 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.578 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.579 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.579 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:03.579 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:03.579 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.579 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.579 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.579 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:03.579 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.580 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:03.580 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:03.580 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.580 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.580 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.580 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:03.580 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:03.580 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:03.580 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 4 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:03.582 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": [ 2024-09-17T15:48:03.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:03.583 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:03.583 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:03.583 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:03.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:03.583 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:03.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:03.583 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:03.583 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:03.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f705cf9b640 1 -- 172.21.15.64:0/2567126730 >> v1:172.21.15.100:6812/4210963484 conn(0x7f7038040d30 legacy=0x7f70380431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f705cf9b640 10 monclient: shutdown 2024-09-17T15:48:03.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f705cf9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:03.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f705cf9b640 1 -- 172.21.15.64:0/2567126730 >> v1:172.21.15.64:6793/0 conn(0x7f70580a54b0 legacy=0x7f70580a4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f7056d9d640 1 -- 172.21.15.64:0/2567126730 reap_dead start 2024-09-17T15:48:03.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f7056d9d640 1 -- 172.21.15.64:0/2567126730 reap_dead start 2024-09-17T15:48:03.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f705cf9b640 1 -- 172.21.15.64:0/2567126730 shutdown_connections 2024-09-17T15:48:03.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f705cf9b640 1 -- 172.21.15.64:0/2567126730 >> 172.21.15.64:0/2567126730 conn(0x7f705806daa0 msgr2=0x7f70580a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:03.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f705cf9b640 1 -- 172.21.15.64:0/2567126730 shutdown_connections 2024-09-17T15:48:03.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.535+0000 7f705cf9b640 1 -- 172.21.15.64:0/2567126730 wait complete. 2024-09-17T15:48:03.585 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:03.586 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:48:03.586 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:48:03.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.684+0000 7fa052041640 10 monclient: get_monmap_and_config 2024-09-17T15:48:03.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.684+0000 7fa052041640 10 monclient: build_initial_monmap 2024-09-17T15:48:03.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.684+0000 7fa052041640 10 monclient: monmap: 2024-09-17T15:48:03.687 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:03.687 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.687 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:03.686163+0000 2024-09-17T15:48:03.687 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:03.686163+0000 2024-09-17T15:48:03.687 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:03.687 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.687 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:03.688 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:03.688 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:03.688 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:03.688 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:03.688 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:03.688 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:03.688 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:03.688 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:03.688 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa052041640 1 Processor -- start 2024-09-17T15:48:03.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa052041640 1 -- start start 2024-09-17T15:48:03.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa052041640 10 monclient: init 2024-09-17T15:48:03.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa052041640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa052041640 10 monclient: _add_conns ranks=[8,7,2,5,4,6,1,0,3] 2024-09-17T15:48:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa052041640 10 monclient(hunting): picked mon.noname-e con 0x7fa04c0a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa052041640 10 monclient(hunting): picked mon.noname-i con 0x7fa04c135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa052041640 10 monclient(hunting): picked mon.noname-b con 0x7fa04c135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa052041640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa04c09d6f0 con 0x7fa04c135f70 2024-09-17T15:48:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa052041640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa04c09d9d0 con 0x7fa04c135b50 2024-09-17T15:48:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa052041640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa04c09dcb0 con 0x7fa04c0a5520 2024-09-17T15:48:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa052041640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa052041640 10 monclient(hunting): authenticate will time out at 881.361669s 2024-09-17T15:48:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa04b7fe640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fa04c0a5520 0x7fa04c0a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:34962/0 (socket says 172.21.15.64:34962) 2024-09-17T15:48:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa04bfff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fa04c135f70 0x7fa04c1415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:33962/0 (socket says 172.21.15.64:33962) 2024-09-17T15:48:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.685+0000 7fa04b7fe640 1 -- 172.21.15.64:0/2454223870 learned_addr learned my addr 172.21.15.64:0/2454223870 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:03.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 1 -- 172.21.15.64:0/2454223870 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2364913514 0 0) 0x7fa04c09d6f0 con 0x7fa04c135f70 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient(hunting): my global_id is 4880 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 1 -- 172.21.15.64:0/2454223870 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa034003520 con 0x7fa04c135f70 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 1 -- 172.21.15.64:0/2454223870 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1655994315 0 0) 0x7fa04c09dcb0 con 0x7fa04c0a5520 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient(hunting): my global_id is 5048 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 1 -- 172.21.15.64:0/2454223870 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa04c09d6f0 con 0x7fa04c0a5520 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 1 -- 172.21.15.64:0/2454223870 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2863385318 0 0) 0x7fa034003520 con 0x7fa04c135f70 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 1 -- 172.21.15.64:0/2454223870 >> v1:172.21.15.64:6793/0 conn(0x7fa04c0a5520 legacy=0x7fa04c0a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 1 -- 172.21.15.64:0/2454223870 >> v1:172.21.15.100:6792/0 conn(0x7fa04c135b50 legacy=0x7fa04c13deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:03.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 1 monclient: found mon.noname-b 2024-09-17T15:48:03.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:03.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:48:03.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 1 -- 172.21.15.64:0/2454223870 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04c09ccd0 con 0x7fa04c135f70 2024-09-17T15:48:03.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:48:03.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:48:03.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:03.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 1 -- 172.21.15.64:0/2454223870 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa044001bb0 con 0x7fa04c135f70 2024-09-17T15:48:03.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa052041640 5 monclient: authenticate success, global_id 4880 2024-09-17T15:48:03.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa052041640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:03.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:03.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient: dump: 2024-09-17T15:48:03.701 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.701 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.701 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.701 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.701 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.701 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.701 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.702 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.702 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.702 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.702 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.702 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.702 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.702 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.702 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.702 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa052041640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:03.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 1 -- 172.21.15.64:0/2454223870 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa044001d50 con 0x7fa04c135f70 2024-09-17T15:48:03.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:03.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 1 -- 172.21.15.64:0/2454223870 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa044004ea0 con 0x7fa04c135f70 2024-09-17T15:48:03.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:03.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa04a7fc640 10 monclient: dump: 2024-09-17T15:48:03.703 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.703 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.703 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.703 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.703 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.703 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.703 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.703 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.703 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.703 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.704 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.704 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.704 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.704 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.704 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.704 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa052041640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:03.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa052041640 4 set_mon_vals no callback set 2024-09-17T15:48:03.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa052041640 10 monclient: shutdown 2024-09-17T15:48:03.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa052041640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:03.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.686+0000 7fa052041640 1 -- 172.21.15.64:0/2454223870 >> v1:172.21.15.64:6790/0 conn(0x7fa04c135f70 legacy=0x7fa04c1415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.687+0000 7fa052041640 1 -- 172.21.15.64:0/2454223870 shutdown_connections 2024-09-17T15:48:03.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.687+0000 7fa052041640 1 -- 172.21.15.64:0/2454223870 >> 172.21.15.64:0/2454223870 conn(0x7fa04c06db30 msgr2=0x7fa04c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:03.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.687+0000 7fa052041640 1 -- 172.21.15.64:0/2454223870 shutdown_connections 2024-09-17T15:48:03.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.687+0000 7fa052041640 1 -- 172.21.15.64:0/2454223870 wait complete. 2024-09-17T15:48:03.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.687+0000 7fa052041640 10 monclient: build_initial_monmap 2024-09-17T15:48:03.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.687+0000 7fa052041640 10 monclient: monmap: 2024-09-17T15:48:03.705 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:03.705 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:03.705 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:03.705 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:03.705 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:03.705 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:03.705 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:03.705 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa052041640 1 Processor -- start 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa052041640 1 -- start start 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa052041640 10 monclient: init 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa052041640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa052041640 10 monclient: _add_conns ranks=[2,5,8,6,3,4,1,7,0] 2024-09-17T15:48:03.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa052041640 10 monclient(hunting): picked mon.noname-b con 0x7fa04c0a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:03.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa052041640 10 monclient(hunting): picked mon.noname-h con 0x7fa04c135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:03.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa052041640 10 monclient(hunting): picked mon.noname-e con 0x7fa04c135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:03.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa052041640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa04c09e470 con 0x7fa04c0a5520 2024-09-17T15:48:03.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa052041640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa04c134ff0 con 0x7fa04c135b50 2024-09-17T15:48:03.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa052041640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa04c0a8cf0 con 0x7fa04c135f70 2024-09-17T15:48:03.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa052041640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:03.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa052041640 10 monclient(hunting): authenticate will time out at 881.364784s 2024-09-17T15:48:03.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa04b7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fa04c0a5520 0x7fa04c131a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:33978/0 (socket says 172.21.15.64:33978) 2024-09-17T15:48:03.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.688+0000 7fa04b7fe640 1 -- 172.21.15.64:0/1494673348 learned_addr learned my addr 172.21.15.64:0/1494673348 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:03.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3718130937 0 0) 0x7fa04c09e470 con 0x7fa04c0a5520 2024-09-17T15:48:03.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): my global_id is 4889 2024-09-17T15:48:03.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa038003520 con 0x7fa04c0a5520 2024-09-17T15:48:03.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3294154240 0 0) 0x7fa04c0a8cf0 con 0x7fa04c135f70 2024-09-17T15:48:03.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): my global_id is 5057 2024-09-17T15:48:03.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa04c09e470 con 0x7fa04c135f70 2024-09-17T15:48:03.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3361576398 0 0) 0x7fa04c134ff0 con 0x7fa04c135b50 2024-09-17T15:48:03.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:03.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:03.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:03.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): my global_id is 114197 2024-09-17T15:48:03.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa04c0a8cf0 con 0x7fa04c135b50 2024-09-17T15:48:03.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 400189279 0 0) 0x7fa038003520 con 0x7fa04c0a5520 2024-09-17T15:48:03.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa04c134ff0 con 0x7fa04c0a5520 2024-09-17T15:48:03.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa03c001460 con 0x7fa04c0a5520 2024-09-17T15:48:03.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:03.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:03.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): dump: 2024-09-17T15:48:03.709 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.709 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.709 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.709 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.709 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.710 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1069940003 0 0) 0x7fa04c09e470 con 0x7fa04c135f70 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa038003520 con 0x7fa04c135f70 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa044001af0 con 0x7fa04c135f70 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): dump: 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.711 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.712 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.712 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.712 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.712 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.712 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.712 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.712 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 412559920 0 0) 0x7fa04c0a8cf0 con 0x7fa04c135b50 2024-09-17T15:48:03.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa04c09e470 con 0x7fa04c135b50 2024-09-17T15:48:03.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa0400019d0 con 0x7fa04c135b50 2024-09-17T15:48:03.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:03.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:48:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient(hunting): dump: 2024-09-17T15:48:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.714 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3835192241 0 0) 0x7fa04c134ff0 con 0x7fa04c0a5520 2024-09-17T15:48:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 >> v1:172.21.15.64:6793/0 conn(0x7fa04c135f70 legacy=0x7fa04c132820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 >> v1:172.21.15.100:6791/0 conn(0x7fa04c135b50 legacy=0x7fa04c132110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient: _finish_hunting 0 2024-09-17T15:48:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 monclient: found mon.b 2024-09-17T15:48:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:03.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04c09ff30 con 0x7fa04c0a5520 2024-09-17T15:48:03.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient: _finish_auth 0 2024-09-17T15:48:03.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient: _check_auth_tickets 2024-09-17T15:48:03.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:03.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa052041640 5 monclient: authenticate success, global_id 4889 2024-09-17T15:48:03.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa052041640 10 monclient: _renew_subs 2024-09-17T15:48:03.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa052041640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:03.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa052041640 1 -- 172.21.15.64:0/1494673348 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa04c09efd0 con 0x7fa04c0a5520 2024-09-17T15:48:03.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa052041640 10 monclient: _renew_subs 2024-09-17T15:48:03.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa052041640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:03.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa052041640 1 -- 172.21.15.64:0/1494673348 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa04c1d7340 con 0x7fa04c0a5520 2024-09-17T15:48:03.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa03c0016e0 con 0x7fa04c0a5520 2024-09-17T15:48:03.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:03.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa03c004dd0 con 0x7fa04c0a5520 2024-09-17T15:48:03.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:03.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:03.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa049ffb640 4 set_mon_vals no callback set 2024-09-17T15:48:03.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.689+0000 7fa033fff640 10 monclient: dump: 2024-09-17T15:48:03.716 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:03.716 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:03.716 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.716 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:03.716 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:03.717 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:03.717 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:03.717 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:03.717 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:03.717 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:03.717 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:03.717 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:03.717 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:03.717 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:03.717 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:03.717 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:03.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.690+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fa03c0116b0 con 0x7fa04c0a5520 2024-09-17T15:48:03.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.690+0000 7fa052041640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:03.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.690+0000 7fa052041640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:03.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.690+0000 7fa052041640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:03.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.690+0000 7fa052041640 1 -- 172.21.15.64:0/1494673348 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa02c005080 con 0x7fa04c0a5520 2024-09-17T15:48:03.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.690+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(109..109 src has 1..109) ==== 3898+0+0 (unknown 1074856076 0 0) 0x7fa03c04ebc0 con 0x7fa04c0a5520 2024-09-17T15:48:03.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.690+0000 7fa04affd640 10 monclient: get_auth_request con 0x7fa038040d50 auth_method 0 2024-09-17T15:48:03.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.695+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fa03c017690 con 0x7fa04c0a5520 2024-09-17T15:48:03.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.695+0000 7fa033fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:03.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.695+0000 7fa033fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:03.857 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36414]: starting. 2024-09-17T15:48:03.857 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36414]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36412 2024-09-17T15:48:03.857 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36414]: shutting down. 2024-09-17T15:48:03.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.963+0000 7fa052041640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:48:03.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.963+0000 7fa052041640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:48:03.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.963+0000 7fa052041640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:03.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.963+0000 7fa052041640 1 -- 172.21.15.64:0/1494673348 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fa02c005c40 con 0x7fa04c0a5520 2024-09-17T15:48:03.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.965+0000 7fa033fff640 1 -- 172.21.15.64:0/1494673348 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fa03c016f90 con 0x7fa04c0a5520 2024-09-17T15:48:03.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.965+0000 7fa033fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:48:03.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.965+0000 7fa033fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:03.973 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.965+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.973 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.965+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=1904497946,config=2586702516,health=3939975164,kv=1246312931,logm=4168676503}) 2024-09-17T15:48:03.973 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.965+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=11,kv=12,logm=58} crc {auth=70473412,config=2586702516,health=926335158,kv=1246312931,logm=1550749941}) 2024-09-17T15:48:03.974 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.965+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.974 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.965+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=1904497946,config=2586702516,health=3939975164,kv=1246312931,logm=4168676503}) 2024-09-17T15:48:03.974 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.965+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=10,kv=10,logm=60} crc {auth=2246466654,config=2586702516,health=2285846686,kv=2761795233,logm=4041954380}) 2024-09-17T15:48:03.974 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.965+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.974 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=1904497946,config=2586702516,health=3939975164,kv=1246312931,logm=4168676503}) 2024-09-17T15:48:03.974 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=18,config=2,health=10,kv=11,logm=59} crc {auth=2588756962,config=2586702516,health=2285846686,kv=4220610509,logm=1229746881}) 2024-09-17T15:48:03.974 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.974 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=1904497946,config=2586702516,health=3939975164,kv=1246312931,logm=4168676503}) 2024-09-17T15:48:03.974 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=2981925337,config=2586702516,health=3939975164,kv=1246312931,logm=2148702330}) 2024-09-17T15:48:03.974 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.974 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=1904497946,config=2586702516,health=3939975164,kv=1246312931,logm=4168676503}) 2024-09-17T15:48:03.974 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=16,config=2,health=11,kv=11,logm=60} crc {auth=1192262779,config=2586702516,health=3785392502,kv=3258717798,logm=728789002}) 2024-09-17T15:48:03.974 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.975 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=1904497946,config=2586702516,health=3939975164,kv=1246312931,logm=4168676503}) 2024-09-17T15:48:03.975 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=17,config=2,health=10,kv=12,logm=59} crc {auth=2981925337,config=2586702516,health=156520775,kv=1246312931,logm=3734481698}) 2024-09-17T15:48:03.975 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.975 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=1904497946,config=2586702516,health=3939975164,kv=1246312931,logm=4168676503}) 2024-09-17T15:48:03.975 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=17,config=2,health=12,kv=11,logm=58} crc {auth=433409455,config=2586702516,health=3939975164,kv=243807300,logm=4067232646}) 2024-09-17T15:48:03.975 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.975 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=1904497946,config=2586702516,health=3939975164,kv=1246312931,logm=4168676503}) 2024-09-17T15:48:03.975 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.966+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=18,config=2,health=11,kv=11,logm=58} crc {auth=2246466654,config=2586702516,health=3425623920,kv=2686754084,logm=2257106625}) 2024-09-17T15:48:03.975 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.975 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201438253}) 2024-09-17T15:48:03.975 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=397344403}) 2024-09-17T15:48:03.975 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.976 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201438253}) 2024-09-17T15:48:03.976 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3406472134}) 2024-09-17T15:48:03.976 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.976 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201438253}) 2024-09-17T15:48:03.976 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3016153274}) 2024-09-17T15:48:03.976 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.976 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201438253}) 2024-09-17T15:48:03.976 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2115669533}) 2024-09-17T15:48:03.976 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.976 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201438253}) 2024-09-17T15:48:03.976 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3340670953}) 2024-09-17T15:48:03.977 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.977 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201438253}) 2024-09-17T15:48:03.977 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2614358194}) 2024-09-17T15:48:03.977 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.977 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201438253}) 2024-09-17T15:48:03.977 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3113194060}) 2024-09-17T15:48:03.977 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.977 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201438253}) 2024-09-17T15:48:03.977 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.967+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1257477192}) 2024-09-17T15:48:03.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.968+0000 7fa052041640 1 -- 172.21.15.64:0/1494673348 >> v1:172.21.15.100:6812/4210963484 conn(0x7fa038040d50 legacy=0x7fa038043210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.968+0000 7fa052041640 10 monclient: shutdown 2024-09-17T15:48:03.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.968+0000 7fa052041640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:03.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.968+0000 7fa052041640 1 -- 172.21.15.64:0/1494673348 >> v1:172.21.15.64:6790/0 conn(0x7fa04c0a5520 legacy=0x7fa04c131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:03.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.968+0000 7fa052041640 1 -- 172.21.15.64:0/1494673348 shutdown_connections 2024-09-17T15:48:03.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.968+0000 7fa052041640 1 -- 172.21.15.64:0/1494673348 >> 172.21.15.64:0/1494673348 conn(0x7fa04c06db30 msgr2=0x7fa04c12c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:03.979 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.968+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.979 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.968+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144726472}) 2024-09-17T15:48:03.979 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.968+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1074074470}) 2024-09-17T15:48:03.980 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.980 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144726472}) 2024-09-17T15:48:03.980 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2897982619}) 2024-09-17T15:48:03.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.969+0000 7fa052041640 1 -- 172.21.15.64:0/1494673348 shutdown_connections 2024-09-17T15:48:03.980 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.980 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144726472}) 2024-09-17T15:48:03.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:03.969+0000 7fa052041640 1 -- 172.21.15.64:0/1494673348 wait complete. 2024-09-17T15:48:03.990 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3684293834}) 2024-09-17T15:48:03.990 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.990 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144726472}) 2024-09-17T15:48:03.990 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2897010402}) 2024-09-17T15:48:03.990 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.990 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144726472}) 2024-09-17T15:48:03.990 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=66952702}) 2024-09-17T15:48:03.990 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144726472}) 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3023751779}) 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144726472}) 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=218513812}) 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144726472}) 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.969+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1586281760}) 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026353741}) 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=864632362}) 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026353741}) 2024-09-17T15:48:03.991 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1488733296}) 2024-09-17T15:48:03.992 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.992 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026353741}) 2024-09-17T15:48:03.992 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4141713814}) 2024-09-17T15:48:03.992 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.992 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026353741}) 2024-09-17T15:48:03.992 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4255350457}) 2024-09-17T15:48:03.992 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.992 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026353741}) 2024-09-17T15:48:03.992 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2053303522}) 2024-09-17T15:48:03.992 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.992 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026353741}) 2024-09-17T15:48:03.992 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1961872014}) 2024-09-17T15:48:03.992 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.993 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026353741}) 2024-09-17T15:48:03.993 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1694977268}) 2024-09-17T15:48:03.993 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.993 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026353741}) 2024-09-17T15:48:03.993 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.970+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3531873194}) 2024-09-17T15:48:03.993 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.993 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406863845}) 2024-09-17T15:48:03.993 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2219555473}) 2024-09-17T15:48:03.993 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.993 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406863845}) 2024-09-17T15:48:03.993 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1142216388}) 2024-09-17T15:48:03.993 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.993 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406863845}) 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=534996890}) 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406863845}) 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1935552926}) 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406863845}) 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2999543934}) 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406863845}) 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2395515640}) 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406863845}) 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3428510534}) 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.994 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406863845}) 2024-09-17T15:48:03.995 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.971+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2308050587}) 2024-09-17T15:48:03.995 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.995 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251981628}) 2024-09-17T15:48:03.995 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2371148923}) 2024-09-17T15:48:03.995 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.995 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251981628}) 2024-09-17T15:48:03.995 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011454309}) 2024-09-17T15:48:03.995 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.995 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251981628}) 2024-09-17T15:48:03.995 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3587567696}) 2024-09-17T15:48:03.995 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.995 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251981628}) 2024-09-17T15:48:03.995 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=797618232}) 2024-09-17T15:48:03.996 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.996 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251981628}) 2024-09-17T15:48:03.996 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.972+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4131672704}) 2024-09-17T15:48:03.996 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.973+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.996 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.973+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251981628}) 2024-09-17T15:48:03.996 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.973+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4250736336}) 2024-09-17T15:48:03.996 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.973+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.996 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.973+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251981628}) 2024-09-17T15:48:03.996 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.973+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2158167980}) 2024-09-17T15:48:03.996 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.973+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.996 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.973+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251981628}) 2024-09-17T15:48:03.996 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.973+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1405875023}) 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1152370599}) 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3221691157}) 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1152370599}) 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788368858}) 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1152370599}) 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2393198846}) 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1152370599}) 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2370030097}) 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.997 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1152370599}) 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4237525874}) 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1152370599}) 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3579145174}) 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1152370599}) 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2548865990}) 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1152370599}) 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.974+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4200894918}) 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10} crc {logm=3151114398,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1005661406,mon_config_key=2997267760}) 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=67,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13} crc {logm=3562873238,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1366525207,mon_config_key=409289127}) 2024-09-17T15:48:03.998 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.999 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10} crc {logm=3151114398,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1005661406,mon_config_key=2997267760}) 2024-09-17T15:48:03.999 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=67,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13} crc {logm=3731086340,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073260,mgr_metadata=4041113850,mgrstat=3874701900,mon_config_key=672978304}) 2024-09-17T15:48:03.999 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.999 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10} crc {logm=3151114398,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1005661406,mon_config_key=2997267760}) 2024-09-17T15:48:03.999 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=67,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13} crc {logm=1302437309,mdsmap=1462844619,mgr=2417410626,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1005661406,mon_config_key=2497640544}) 2024-09-17T15:48:03.999 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.999 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10} crc {logm=3151114398,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1005661406,mon_config_key=2997267760}) 2024-09-17T15:48:03.999 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=8,mon_config_key=12} crc {logm=3988988439,mdsmap=1462844619,mgr=4276097997,mgr_command_descs=1957073259,mgrstat=1005661406,mon_config_key=2306131471}) 2024-09-17T15:48:03.999 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.999 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10} crc {logm=3151114398,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1005661406,mon_config_key=2997267760}) 2024-09-17T15:48:03.999 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=14} crc {logm=104083561,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1005661406,mon_config_key=641472809}) 2024-09-17T15:48:03.999 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:03.999 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10} crc {logm=3151114398,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1005661406,mon_config_key=2997267760}) 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=67,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=12} crc {logm=2222815530,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=3133490957,mon_config_key=2306131471}) 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10} crc {logm=3151114398,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1005661406,mon_config_key=2997267760}) 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13} crc {logm=864591944,mdsmap=1462844619,mgr=1860383721,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1005661406,mon_config_key=409289127}) 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10} crc {logm=3151114398,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1005661406,mon_config_key=2997267760}) 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.975+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11} crc {logm=1294308182,mdsmap=1462844619,mgr=2970304559,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1005661406,mon_config_key=2802973941}) 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=81} crc {mon_config_key=4125316642,monmap=1573330833,nvmeofgw=1596038039,osd_metadata=4100752439,osd_pg_creating=1659734222,osd_snap=2941457380,osdmap=2496343237}) 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=6,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=80} crc {mon_config_key=154554388,monmap=3140514009,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=1659734222,osd_snap=723727393,osdmap=1983551075}) 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=81} crc {mon_config_key=4125316642,monmap=1573330833,nvmeofgw=1596038039,osd_metadata=4100752439,osd_pg_creating=1659734222,osd_snap=2941457380,osdmap=2496343237}) 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=80} crc {mon_config_key=4125316642,monmap=1573330833,nvmeofgw=598337686,osd_metadata=60438592,osd_pg_creating=1659734222,osd_snap=723727393,osdmap=1113852570}) 2024-09-17T15:48:04.000 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.001 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=81} crc {mon_config_key=4125316642,monmap=1573330833,nvmeofgw=1596038039,osd_metadata=4100752439,osd_pg_creating=1659734222,osd_snap=2941457380,osdmap=2496343237}) 2024-09-17T15:48:04.001 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=7,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=78} crc {mon_config_key=4051167929,monmap=1573330833,nvmeofgw=598337686,osd_metadata=1169256444,osd_pg_creating=1659734222,osd_snap=723727393,osdmap=3771660908}) 2024-09-17T15:48:04.001 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.001 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=81} crc {mon_config_key=4125316642,monmap=1573330833,nvmeofgw=1596038039,osd_metadata=4100752439,osd_pg_creating=1659734222,osd_snap=2941457380,osdmap=2496343237}) 2024-09-17T15:48:04.001 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {mon_config_key=7,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=78} crc {mon_config_key=4051167929,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=1659734222,osd_snap=723727393,osdmap=1256821581}) 2024-09-17T15:48:04.001 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.001 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=81} crc {mon_config_key=4125316642,monmap=1573330833,nvmeofgw=1596038039,osd_metadata=4100752439,osd_pg_creating=1659734222,osd_snap=2941457380,osdmap=2496343237}) 2024-09-17T15:48:04.001 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {mon_config_key=7,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=80} crc {mon_config_key=4051167929,monmap=1573330833,nvmeofgw=3140514009,osd_metadata=4100752439,osd_pg_creating=1659734222,osd_snap=2941457380,osdmap=4194983788}) 2024-09-17T15:48:04.001 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.001 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=81} crc {mon_config_key=4125316642,monmap=1573330833,nvmeofgw=1596038039,osd_metadata=4100752439,osd_pg_creating=1659734222,osd_snap=2941457380,osdmap=2496343237}) 2024-09-17T15:48:04.001 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=80} crc {mon_config_key=3705742558,monmap=1573330833,nvmeofgw=598337686,osd_metadata=179057771,osd_pg_creating=1659734222,osd_snap=723727393,osdmap=400938907}) 2024-09-17T15:48:04.001 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.001 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=81} crc {mon_config_key=4125316642,monmap=1573330833,nvmeofgw=1596038039,osd_metadata=4100752439,osd_pg_creating=1659734222,osd_snap=2941457380,osdmap=2496343237}) 2024-09-17T15:48:04.002 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=7,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=78} crc {mon_config_key=4051167929,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=1659734222,osd_snap=723727393,osdmap=1034710171}) 2024-09-17T15:48:04.002 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.002 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=81} crc {mon_config_key=4125316642,monmap=1573330833,nvmeofgw=1596038039,osd_metadata=4100752439,osd_pg_creating=1659734222,osd_snap=2941457380,osdmap=2496343237}) 2024-09-17T15:48:04.002 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.977+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=7,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=78} crc {mon_config_key=4051167929,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=1659734222,osd_snap=723727393,osdmap=4140292412}) 2024-09-17T15:48:04.002 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.002 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=3579472394}) 2024-09-17T15:48:04.002 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=97} crc {osdmap=360909794}) 2024-09-17T15:48:04.002 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.002 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=3579472394}) 2024-09-17T15:48:04.002 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=88} crc {osdmap=2021654811}) 2024-09-17T15:48:04.002 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.002 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=3579472394}) 2024-09-17T15:48:04.003 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=94} crc {osdmap=3760507828}) 2024-09-17T15:48:04.003 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.003 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=3579472394}) 2024-09-17T15:48:04.003 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=94} crc {osdmap=3613122331}) 2024-09-17T15:48:04.003 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.003 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=3579472394}) 2024-09-17T15:48:04.003 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=95} crc {osdmap=2811291773}) 2024-09-17T15:48:04.003 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.003 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=3579472394}) 2024-09-17T15:48:04.003 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=95} crc {osdmap=3825613488}) 2024-09-17T15:48:04.003 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.003 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=3579472394}) 2024-09-17T15:48:04.003 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=96} crc {osdmap=1849720783}) 2024-09-17T15:48:04.004 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:04.004 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.979+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=3579472394}) 2024-09-17T15:48:04.004 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-09-17T15:48:03.980+0000 7f0ba85db640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=95} crc {osdmap=1775995495}) 2024-09-17T15:48:04.004 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:48:05.006 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-17T15:48:05.006 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:48:05.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.113+0000 7f64b5bf5640 10 monclient: get_monmap_and_config 2024-09-17T15:48:05.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.113+0000 7f64b5bf5640 10 monclient: build_initial_monmap 2024-09-17T15:48:05.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.113+0000 7f64b5bf5640 10 monclient: monmap: 2024-09-17T15:48:05.117 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:05.117 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.117 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:05.115637+0000 2024-09-17T15:48:05.117 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:05.115637+0000 2024-09-17T15:48:05.117 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:05.117 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.117 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:05.117 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:05.117 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:05.117 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:05.117 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:05.117 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:05.117 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:05.118 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:05.118 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:05.118 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.114+0000 7f64b5bf5640 1 Processor -- start 2024-09-17T15:48:05.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.114+0000 7f64b5bf5640 1 -- start start 2024-09-17T15:48:05.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.114+0000 7f64b5bf5640 10 monclient: init 2024-09-17T15:48:05.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64b5bf5640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:05.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64b5bf5640 10 monclient: _add_conns ranks=[8,2,0,4,5,1,7,3,6] 2024-09-17T15:48:05.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64b5bf5640 10 monclient(hunting): picked mon.noname-e con 0x7f64b00a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:05.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64b5bf5640 10 monclient(hunting): picked mon.noname-b con 0x7f64b0135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:05.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64b5bf5640 10 monclient(hunting): picked mon.noname-a con 0x7f64b0135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:05.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64b5bf5640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b009d6f0 con 0x7f64b0135f70 2024-09-17T15:48:05.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64b5bf5640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b009d9d0 con 0x7f64b0135b50 2024-09-17T15:48:05.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64b5bf5640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b009dcb0 con 0x7f64b00a5520 2024-09-17T15:48:05.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64b5bf5640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:05.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64b5bf5640 10 monclient(hunting): authenticate will time out at 882.791182s 2024-09-17T15:48:05.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64aeffd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f64b0135b50 0x7f64b013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:52964/0 (socket says 172.21.15.64:52964) 2024-09-17T15:48:05.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64affff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f64b0135f70 0x7f64b01415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:39816/0 (socket says 172.21.15.64:39816) 2024-09-17T15:48:05.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64af7fe640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f64b00a5520 0x7f64b00a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38688/0 (socket says 172.21.15.64:38688) 2024-09-17T15:48:05.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64aeffd640 1 -- 172.21.15.64:0/2452619605 learned_addr learned my addr 172.21.15.64:0/2452619605 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:05.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 1 -- 172.21.15.64:0/2452619605 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1091362201 0 0) 0x7f64b009d6f0 con 0x7f64b0135f70 2024-09-17T15:48:05.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient(hunting): my global_id is 5094 2024-09-17T15:48:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 1 -- 172.21.15.64:0/2452619605 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6498003520 con 0x7f64b0135f70 2024-09-17T15:48:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 1 -- 172.21.15.64:0/2452619605 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 393122194 0 0) 0x7f64b009dcb0 con 0x7f64b00a5520 2024-09-17T15:48:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient(hunting): my global_id is 5102 2024-09-17T15:48:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 1 -- 172.21.15.64:0/2452619605 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64b009d6f0 con 0x7f64b00a5520 2024-09-17T15:48:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 1 -- 172.21.15.64:0/2452619605 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3426281203 0 0) 0x7f64b009d9d0 con 0x7f64b0135b50 2024-09-17T15:48:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient(hunting): my global_id is 4916 2024-09-17T15:48:05.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 1 -- 172.21.15.64:0/2452619605 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64b009dcb0 con 0x7f64b0135b50 2024-09-17T15:48:05.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 1 -- 172.21.15.64:0/2452619605 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2882407698 0 0) 0x7f64b009d6f0 con 0x7f64b00a5520 2024-09-17T15:48:05.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 1 -- 172.21.15.64:0/2452619605 >> v1:172.21.15.64:6790/0 conn(0x7f64b0135b50 legacy=0x7f64b013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 1 -- 172.21.15.64:0/2452619605 >> v1:172.21.15.64:6789/0 conn(0x7f64b0135f70 legacy=0x7f64b01415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:05.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 1 monclient: found mon.noname-e 2024-09-17T15:48:05.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:05.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:48:05.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 1 -- 172.21.15.64:0/2452619605 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b009ccd0 con 0x7f64b00a5520 2024-09-17T15:48:05.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient: _finish_auth 0 2024-09-17T15:48:05.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:48:05.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.115+0000 7f64ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:05.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64b5bf5640 5 monclient: authenticate success, global_id 5102 2024-09-17T15:48:05.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64b5bf5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:05.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64ae7fc640 1 -- 172.21.15.64:0/2452619605 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f64a0001810 con 0x7f64b00a5520 2024-09-17T15:48:05.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64ae7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:48:05.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64ae7fc640 10 monclient: dump: 2024-09-17T15:48:05.129 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.129 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.129 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.129 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.129 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.129 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.129 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.129 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64b5bf5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64ae7fc640 1 -- 172.21.15.64:0/2452619605 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f64a0001a30 con 0x7f64b00a5520 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64ae7fc640 1 -- 172.21.15.64:0/2452619605 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f64a0004b20 con 0x7f64b00a5520 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64ae7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:05.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64ae7fc640 10 monclient: dump: 2024-09-17T15:48:05.131 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.131 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.131 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.131 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.131 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.131 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.131 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.131 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.131 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.131 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.131 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.131 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.132 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.132 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.132 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.132 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64b5bf5640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:05.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64b5bf5640 4 set_mon_vals no callback set 2024-09-17T15:48:05.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64b5bf5640 10 monclient: shutdown 2024-09-17T15:48:05.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64b5bf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:05.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64b5bf5640 1 -- 172.21.15.64:0/2452619605 >> v1:172.21.15.64:6793/0 conn(0x7f64b00a5520 legacy=0x7f64b00a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64b5bf5640 1 -- 172.21.15.64:0/2452619605 shutdown_connections 2024-09-17T15:48:05.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64b5bf5640 1 -- 172.21.15.64:0/2452619605 >> 172.21.15.64:0/2452619605 conn(0x7f64b006db30 msgr2=0x7f64b00ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:05.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64b5bf5640 1 -- 172.21.15.64:0/2452619605 shutdown_connections 2024-09-17T15:48:05.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.116+0000 7f64b5bf5640 1 -- 172.21.15.64:0/2452619605 wait complete. 2024-09-17T15:48:05.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.117+0000 7f64b5bf5640 10 monclient: build_initial_monmap 2024-09-17T15:48:05.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.117+0000 7f64b5bf5640 10 monclient: monmap: 2024-09-17T15:48:05.133 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:05.133 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:05.136 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:05.136 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:05.136 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:05.136 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:05.136 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:05.136 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:05.136 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:05.136 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.117+0000 7f64b5bf5640 1 Processor -- start 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.117+0000 7f64b5bf5640 1 -- start start 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.117+0000 7f64b5bf5640 10 monclient: init 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f64b5bf5640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f64b5bf5640 10 monclient: _add_conns ranks=[4,2,3,7,8,6,0,1,5] 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f64b5bf5640 10 monclient(hunting): picked mon.noname-c con 0x7f64b00a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f64b5bf5640 10 monclient(hunting): picked mon.noname-b con 0x7f64b0135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:05.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f64b5bf5640 10 monclient(hunting): picked mon.noname-g con 0x7f64b0135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f64b5bf5640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b0135060 con 0x7f64b0135b50 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f64b5bf5640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b00a8d70 con 0x7f64b0135f70 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f64b5bf5640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b01ce590 con 0x7f64b00a5520 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f64b5bf5640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f64b5bf5640 10 monclient(hunting): authenticate will time out at 882.794416s 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f64aeffd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f64b0135b50 0x7f64b01cde80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:52966/0 (socket says 172.21.15.64:52966) 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f64aeffd640 1 -- 172.21.15.64:0/1269300624 learned_addr learned my addr 172.21.15.64:0/1269300624 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3078899102 0 0) 0x7f64b0135060 con 0x7f64b0135b50 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 10 monclient(hunting): my global_id is 4925 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f649c003520 con 0x7f64b0135b50 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2017214676 0 0) 0x7f64b01ce590 con 0x7f64b00a5520 2024-09-17T15:48:05.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 10 monclient(hunting): my global_id is 5161 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64b0135060 con 0x7f64b00a5520 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2321643918 0 0) 0x7f649c003520 con 0x7f64b0135b50 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64b01ce590 con 0x7f64b0135b50 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f64a4001420 con 0x7f64b0135b50 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.118+0000 7f6497fff640 10 monclient(hunting): dump: 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.139 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3495415778 0 0) 0x7f64b0135060 con 0x7f64b00a5520 2024-09-17T15:48:05.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f649c003520 con 0x7f64b00a5520 2024-09-17T15:48:05.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f64a00017d0 con 0x7f64b00a5520 2024-09-17T15:48:05.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:05.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:05.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient(hunting): dump: 2024-09-17T15:48:05.141 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.141 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.141 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.141 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.141 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.141 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.141 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.141 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.141 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 271681771 0 0) 0x7f64b00a8d70 con 0x7f64b0135f70 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient(hunting): my global_id is 114186 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64b0135060 con 0x7f64b0135f70 2024-09-17T15:48:05.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3243213328 0 0) 0x7f64b01ce590 con 0x7f64b0135b50 2024-09-17T15:48:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 >> v1:172.21.15.64:6791/0 conn(0x7f64b00a5520 legacy=0x7f64b01cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 >> v1:172.21.15.100:6790/0 conn(0x7f64b0135f70 legacy=0x7f64b01d5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient: _finish_hunting 0 2024-09-17T15:48:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 1 monclient: found mon.b 2024-09-17T15:48:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b009f040 con 0x7f64b0135b50 2024-09-17T15:48:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient: _finish_auth 0 2024-09-17T15:48:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient: _check_auth_tickets 2024-09-17T15:48:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f64b5bf5640 5 monclient: authenticate success, global_id 4925 2024-09-17T15:48:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f64b5bf5640 10 monclient: _renew_subs 2024-09-17T15:48:05.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f64b5bf5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f64b5bf5640 1 -- 172.21.15.64:0/1269300624 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f64b009fe20 con 0x7f64b0135b50 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f64b5bf5640 10 monclient: _renew_subs 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f64b5bf5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f64b5bf5640 1 -- 172.21.15.64:0/1269300624 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f64b00a03e0 con 0x7f64b0135b50 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f64a4004a90 con 0x7f64b0135b50 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f64a4004f80 con 0x7f64b0135b50 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f64adffb640 4 set_mon_vals no callback set 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.119+0000 7f6497fff640 10 monclient: dump: 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.144 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.120+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f64a4011860 con 0x7f64b0135b50 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.120+0000 7f64b5bf5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:05.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.120+0000 7f64b5bf5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:05.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.120+0000 7f64b5bf5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:05.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.120+0000 7f64b5bf5640 1 -- 172.21.15.64:0/1269300624 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6490005080 con 0x7f64b0135b50 2024-09-17T15:48:05.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.120+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(111..111 src has 1..111) ==== 3898+0+0 (unknown 2549879297 0 0) 0x7f64a404eb10 con 0x7f64b0135b50 2024-09-17T15:48:05.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.120+0000 7f64af7fe640 10 monclient: get_auth_request con 0x7f649c040ac0 auth_method 0 2024-09-17T15:48:05.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.125+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f64a4017760 con 0x7f64b0135b50 2024-09-17T15:48:05.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.125+0000 7f6497fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:05.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.125+0000 7f6497fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:05.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.385+0000 7f64b5bf5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:48:05.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.385+0000 7f64b5bf5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:05.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.385+0000 7f64b5bf5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:05.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.385+0000 7f64b5bf5640 1 -- 172.21.15.64:0/1269300624 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6490005370 con 0x7f64b0135b50 2024-09-17T15:48:05.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.386+0000 7f6497fff640 1 -- 172.21.15.64:0/1269300624 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 4026074670) 0x7f64a4017060 con 0x7f64b0135b50 2024-09-17T15:48:05.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.386+0000 7f6497fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:05.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.386+0000 7f6497fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:05.389 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":16,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:48:05.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.388+0000 7f64b5bf5640 1 -- 172.21.15.64:0/1269300624 >> v1:172.21.15.100:6812/4210963484 conn(0x7f649c040ac0 legacy=0x7f649c042f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.388+0000 7f64b5bf5640 10 monclient: shutdown 2024-09-17T15:48:05.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.388+0000 7f64b5bf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:05.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.388+0000 7f64b5bf5640 1 -- 172.21.15.64:0/1269300624 >> v1:172.21.15.64:6790/0 conn(0x7f64b0135b50 legacy=0x7f64b01cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.388+0000 7f64b5bf5640 1 -- 172.21.15.64:0/1269300624 shutdown_connections 2024-09-17T15:48:05.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.388+0000 7f64b5bf5640 1 -- 172.21.15.64:0/1269300624 >> 172.21.15.64:0/1269300624 conn(0x7f64b006db30 msgr2=0x7f64b00a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:05.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.388+0000 7f64b5bf5640 1 -- 172.21.15.64:0/1269300624 shutdown_connections 2024-09-17T15:48:05.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.388+0000 7f64b5bf5640 1 -- 172.21.15.64:0/1269300624 wait complete. 2024-09-17T15:48:05.401 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-17T15:48:05.401 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:48:05.401 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-17T15:48:05.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:48:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.500+0000 7fc1f4808640 10 monclient: get_monmap_and_config 2024-09-17T15:48:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.500+0000 7fc1f4808640 10 monclient: build_initial_monmap 2024-09-17T15:48:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.500+0000 7fc1f4808640 10 monclient: monmap: 2024-09-17T15:48:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:05.502440+0000 2024-09-17T15:48:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:05.502440+0000 2024-09-17T15:48:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:05.504 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:05.504 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1f4808640 1 Processor -- start 2024-09-17T15:48:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1f4808640 1 -- start start 2024-09-17T15:48:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1f4808640 10 monclient: init 2024-09-17T15:48:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1f4808640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1f4808640 10 monclient: _add_conns ranks=[1,6,7,0,3,4,8,5,2] 2024-09-17T15:48:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1f4808640 10 monclient(hunting): picked mon.noname-f con 0x7fc1f00a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:05.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1f4808640 10 monclient(hunting): picked mon.noname-d con 0x7fc1f0135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1f4808640 10 monclient(hunting): picked mon.noname-i con 0x7fc1f0135f70 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1f4808640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f009d6f0 con 0x7fc1f00a5520 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1f4808640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f009d9d0 con 0x7fc1f0135b50 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1f4808640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f009dcb0 con 0x7fc1f0135f70 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1f4808640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1f4808640 10 monclient(hunting): authenticate will time out at 883.177849s 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1e7fff640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fc1f0135b50 0x7fc1f013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:50296/0 (socket says 172.21.15.64:50296) 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.501+0000 7fc1e7fff640 1 -- 172.21.15.64:0/3162930989 learned_addr learned my addr 172.21.15.64:0/3162930989 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 -- 172.21.15.64:0/3162930989 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 83245758 0 0) 0x7fc1f009d9d0 con 0x7fc1f0135b50 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient(hunting): my global_id is 114189 2024-09-17T15:48:05.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 -- 172.21.15.64:0/3162930989 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1dc003520 con 0x7fc1f0135b50 2024-09-17T15:48:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 -- 172.21.15.64:0/3162930989 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1563916898 0 0) 0x7fc1f009d6f0 con 0x7fc1f00a5520 2024-09-17T15:48:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient(hunting): my global_id is 5221 2024-09-17T15:48:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 -- 172.21.15.64:0/3162930989 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1f009d9d0 con 0x7fc1f00a5520 2024-09-17T15:48:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 -- 172.21.15.64:0/3162930989 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 418083425 0 0) 0x7fc1f009dcb0 con 0x7fc1f0135f70 2024-09-17T15:48:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient(hunting): my global_id is 5056 2024-09-17T15:48:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 -- 172.21.15.64:0/3162930989 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1f009d6f0 con 0x7fc1f0135f70 2024-09-17T15:48:05.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 -- 172.21.15.64:0/3162930989 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3370586677 0 0) 0x7fc1dc003520 con 0x7fc1f0135b50 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 -- 172.21.15.64:0/3162930989 >> v1:172.21.15.100:6792/0 conn(0x7fc1f0135f70 legacy=0x7fc1f01415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 -- 172.21.15.64:0/3162930989 >> v1:172.21.15.100:6789/0 conn(0x7fc1f00a5520 legacy=0x7fc1f00a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 monclient: found mon.noname-d 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 -- 172.21.15.64:0/3162930989 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f009ccd0 con 0x7fc1f0135b50 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient: _finish_auth 0 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient: _check_auth_tickets 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 -- 172.21.15.64:0/3162930989 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc1d80017b0 con 0x7fc1f0135b50 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1f4808640 5 monclient: authenticate success, global_id 114189 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1f4808640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:05.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient: dump: 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.517 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.518 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1f4808640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 -- 172.21.15.64:0/3162930989 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1d8001950 con 0x7fc1f0135b50 2024-09-17T15:48:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1f4808640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 1 -- 172.21.15.64:0/3162930989 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc1d8004be0 con 0x7fc1f0135b50 2024-09-17T15:48:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1f4808640 4 set_mon_vals no callback set 2024-09-17T15:48:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:48:05.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.502+0000 7fc1edd9b640 10 monclient: dump: 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.519 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.520 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.503+0000 7fc1f4808640 10 monclient: shutdown 2024-09-17T15:48:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.503+0000 7fc1f4808640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.503+0000 7fc1f4808640 1 -- 172.21.15.64:0/3162930989 >> v1:172.21.15.64:6792/0 conn(0x7fc1f0135b50 legacy=0x7fc1f013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.503+0000 7fc1f4808640 1 -- 172.21.15.64:0/3162930989 shutdown_connections 2024-09-17T15:48:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.503+0000 7fc1f4808640 1 -- 172.21.15.64:0/3162930989 >> 172.21.15.64:0/3162930989 conn(0x7fc1f006db30 msgr2=0x7fc1f00ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.503+0000 7fc1f4808640 1 -- 172.21.15.64:0/3162930989 shutdown_connections 2024-09-17T15:48:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.503+0000 7fc1f4808640 1 -- 172.21.15.64:0/3162930989 wait complete. 2024-09-17T15:48:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.503+0000 7fc1f4808640 10 monclient: build_initial_monmap 2024-09-17T15:48:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.503+0000 7fc1f4808640 10 monclient: monmap: 2024-09-17T15:48:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:05.521 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1f4808640 1 Processor -- start 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1f4808640 1 -- start start 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1f4808640 10 monclient: init 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1f4808640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1f4808640 10 monclient: _add_conns ranks=[2,8,6,4,1,5,7,0,3] 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1f4808640 10 monclient(hunting): picked mon.noname-b con 0x7fc1f00a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1f4808640 10 monclient(hunting): picked mon.noname-e con 0x7fc1f0135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1f4808640 10 monclient(hunting): picked mon.noname-d con 0x7fc1f0135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1f4808640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f013b280 con 0x7fc1f00a5520 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1f4808640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f0134840 con 0x7fc1f0135f70 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1f4808640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f0134db0 con 0x7fc1f0135b50 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1f4808640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:05.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1f4808640 10 monclient(hunting): authenticate will time out at 883.180680s 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1ee59c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc1f00a5520 0x7fc1f0132370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:52976/0 (socket says 172.21.15.64:52976) 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1ee59c640 1 -- 172.21.15.64:0/4202059067 learned_addr learned my addr 172.21.15.64:0/4202059067 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.504+0000 7fc1e7fff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fc1f0135b50 0x7fc1f0132a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38690/0 (socket says 172.21.15.64:38690) 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1502707920 0 0) 0x7fc1f0134db0 con 0x7fc1f0135b50 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): my global_id is 5111 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1d0003520 con 0x7fc1f0135b50 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1795190352 0 0) 0x7fc1f0134840 con 0x7fc1f0135f70 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): my global_id is 114198 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1f0134db0 con 0x7fc1f0135f70 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 332301577 0 0) 0x7fc1f013b280 con 0x7fc1f00a5520 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): my global_id is 4934 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1f0134840 con 0x7fc1f00a5520 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 999836427 0 0) 0x7fc1f0134db0 con 0x7fc1f0135f70 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1f013b280 con 0x7fc1f0135f70 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc1e8001c70 con 0x7fc1f0135f70 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): dump: 2024-09-17T15:48:05.524 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.525 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.526 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1707099206 0 0) 0x7fc1f0134840 con 0x7fc1f00a5520 2024-09-17T15:48:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1f0134db0 con 0x7fc1f00a5520 2024-09-17T15:48:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1960919253 0 0) 0x7fc1d0003520 con 0x7fc1f0135b50 2024-09-17T15:48:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1f0134840 con 0x7fc1f0135b50 2024-09-17T15:48:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc1e00019a0 con 0x7fc1f00a5520 2024-09-17T15:48:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): dump: 2024-09-17T15:48:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.526 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.527 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc1d80016f0 con 0x7fc1f0135b50 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient(hunting): dump: 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.528 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.529 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.529 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.529 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.529 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.529 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.529 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1036887588 0 0) 0x7fc1f0134db0 con 0x7fc1f00a5520 2024-09-17T15:48:05.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 >> v1:172.21.15.64:6793/0 conn(0x7fc1f0135b50 legacy=0x7fc1f0132a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 >> v1:172.21.15.64:6792/0 conn(0x7fc1f0135f70 legacy=0x7fc1f01d6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 monclient: found mon.b 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f00a64b0 con 0x7fc1f00a5520 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1f4808640 5 monclient: authenticate success, global_id 4934 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1f4808640 10 monclient: _renew_subs 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1f4808640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1f4808640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc1f013afa0 con 0x7fc1f00a5520 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1f4808640 10 monclient: _renew_subs 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1f4808640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:05.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1f4808640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc1f01385d0 con 0x7fc1f00a5520 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1e0001ce0 con 0x7fc1f00a5520 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc1e0003c00 con 0x7fc1f00a5520 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1e6ffd640 10 monclient: dump: 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.531 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.505+0000 7fc1ed59a640 4 set_mon_vals no callback set 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.506+0000 7fc1f4808640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.506+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fc1e0003da0 con 0x7fc1f00a5520 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.506+0000 7fc1f4808640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc1d40065c0 con 0x7fc1d4002860 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.506+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(111..111 src has 1..111) ==== 3898+0+0 (unknown 2549879297 0 0) 0x7fc1e004ea30 con 0x7fc1f00a5520 2024-09-17T15:48:05.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.506+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4006728686 0 0) 0x7fc1d40065c0 con 0x7fc1d4002860 2024-09-17T15:48:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.506+0000 7fc1e6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.506+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.506+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.506+0000 7fc1e6ffd640 10 monclient(hunting): my global_id is 5103 2024-09-17T15:48:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.506+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1f0134db0 con 0x7fc1d4002860 2024-09-17T15:48:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.506+0000 7fc1eed9d640 10 monclient: get_auth_request con 0x7fc1d0040c80 auth_method 0 2024-09-17T15:48:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.507+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2151501886 0 0) 0x7fc1f0134db0 con 0x7fc1d4002860 2024-09-17T15:48:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.507+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1d40065c0 con 0x7fc1d4002860 2024-09-17T15:48:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.507+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc1d8004b20 con 0x7fc1d4002860 2024-09-17T15:48:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.507+0000 7fc1e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.507+0000 7fc1e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.507+0000 7fc1e6ffd640 10 monclient: dump: 2024-09-17T15:48:05.533 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.534 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.507+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3567294505 0 0) 0x7fc1d40065c0 con 0x7fc1d4002860 2024-09-17T15:48:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.507+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc1d400a160 con 0x7fc1d4002860 2024-09-17T15:48:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.507+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc1d8007480 con 0x7fc1d4002860 2024-09-17T15:48:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.507+0000 7fc1e6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.507+0000 7fc1e6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.507+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 >> v1:172.21.15.64:6789/0 conn(0x7fc1d4002860 legacy=0x7fc1d4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.507+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 >> v1:172.21.15.64:6789/0 conn(0x7fc1d4002860 legacy=0x7fc1d4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:05.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.513+0000 7fc1f4808640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:05.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.513+0000 7fc1f4808640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc1d4009e20 con 0x7fc1d40090e0 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2996344055 0 0) 0x7fc1d4009e20 con 0x7fc1d40090e0 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 10 monclient(hunting): my global_id is 5112 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1d40065c0 con 0x7fc1d40090e0 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4102041872 0 0) 0x7fc1d40065c0 con 0x7fc1d40090e0 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1d4009e20 con 0x7fc1d40090e0 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc1d8005630 con 0x7fc1d40090e0 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 10 monclient: dump: 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.537 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.538 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2731392207 0 0) 0x7fc1d4009e20 con 0x7fc1d40090e0 2024-09-17T15:48:05.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 --> v1:172.21.15.64:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc1d400c3f0 con 0x7fc1d40090e0 2024-09-17T15:48:05.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7713 (unknown 0 0 4030454924) 0x7fc1d8006b60 con 0x7fc1d40090e0 2024-09-17T15:48:05.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:05.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:05.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 >> v1:172.21.15.64:6789/0 conn(0x7fc1d40090e0 legacy=0x7fc1d4009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.514+0000 7fc1e6ffd640 1 -- 172.21.15.64:0/4202059067 >> v1:172.21.15.64:6789/0 conn(0x7fc1d40090e0 legacy=0x7fc1d4009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:05.539 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:05.539 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:05.539 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:05.539 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "leader", 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 142168, 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.540 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 3, 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:05.541 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:05.542 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:05.543 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:05.544 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:05.544 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:05.544 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:05.544 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:05.544 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:05.544 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:05.544 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:05.544 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:05.544 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:05.544 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:05.544 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:05.544 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:05.544 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:05.545 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.546 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:05.548 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.550 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:05.551 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:05.553 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.554 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.555 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:05.556 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:05.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.558 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:05.558 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:05.558 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:05.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.558 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.558 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:05.558 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:05.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.515+0000 7fc1f4808640 1 -- 172.21.15.64:0/4202059067 >> v1:172.21.15.100:6812/4210963484 conn(0x7fc1d0040c80 legacy=0x7fc1d0043140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.515+0000 7fc1f4808640 10 monclient: shutdown 2024-09-17T15:48:05.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.515+0000 7fc1f4808640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:05.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.515+0000 7fc1f4808640 1 -- 172.21.15.64:0/4202059067 >> v1:172.21.15.64:6790/0 conn(0x7fc1f00a5520 legacy=0x7fc1f0132370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.515+0000 7fc1eed9d640 1 -- 172.21.15.64:0/4202059067 reap_dead start 2024-09-17T15:48:05.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.515+0000 7fc1eed9d640 1 -- 172.21.15.64:0/4202059067 reap_dead start 2024-09-17T15:48:05.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.515+0000 7fc1f4808640 1 -- 172.21.15.64:0/4202059067 shutdown_connections 2024-09-17T15:48:05.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.515+0000 7fc1f4808640 1 -- 172.21.15.64:0/4202059067 >> 172.21.15.64:0/4202059067 conn(0x7fc1f006db30 msgr2=0x7fc1f01340c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:05.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.515+0000 7fc1f4808640 1 -- 172.21.15.64:0/4202059067 shutdown_connections 2024-09-17T15:48:05.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.515+0000 7fc1f4808640 1 -- 172.21.15.64:0/4202059067 wait complete. 2024-09-17T15:48:05.560 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:05.560 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-17T15:48:05.560 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:48:05.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.660+0000 7f2d6fad6640 10 monclient: get_monmap_and_config 2024-09-17T15:48:05.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.660+0000 7f2d6fad6640 10 monclient: build_initial_monmap 2024-09-17T15:48:05.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.660+0000 7f2d6fad6640 10 monclient: monmap: 2024-09-17T15:48:05.664 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:05.664 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.664 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:05.662569+0000 2024-09-17T15:48:05.664 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:05.662569+0000 2024-09-17T15:48:05.664 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:05.664 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.664 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:05.664 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:05.664 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:05.664 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:05.665 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:05.665 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:05.665 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:05.665 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:05.665 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:05.665 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.661+0000 7f2d6fad6640 1 Processor -- start 2024-09-17T15:48:05.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.661+0000 7f2d6fad6640 1 -- start start 2024-09-17T15:48:05.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.661+0000 7f2d6fad6640 10 monclient: init 2024-09-17T15:48:05.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.661+0000 7f2d6fad6640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:05.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.661+0000 7f2d6fad6640 10 monclient: _add_conns ranks=[3,2,6,4,7,5,1,8,0] 2024-09-17T15:48:05.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.661+0000 7f2d6fad6640 10 monclient(hunting): picked mon.noname-g con 0x7f2d680a5520 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:05.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.661+0000 7f2d6fad6640 10 monclient(hunting): picked mon.noname-b con 0x7f2d68135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:05.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6fad6640 10 monclient(hunting): picked mon.noname-d con 0x7f2d68135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:05.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6fad6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d6809d6f0 con 0x7f2d68135b50 2024-09-17T15:48:05.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6fad6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d6809d9d0 con 0x7f2d680a5520 2024-09-17T15:48:05.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6fad6640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d6809dcb0 con 0x7f2d68135f70 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6fad6640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6fad6640 10 monclient(hunting): authenticate will time out at 883.338044s 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6e073640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f2d68135f70 0x7f2d681415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:50312/0 (socket says 172.21.15.64:50312) 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6d071640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f2d68135b50 0x7f2d6813deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:52988/0 (socket says 172.21.15.64:52988) 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6e073640 1 -- 172.21.15.64:0/1229331572 learned_addr learned my addr 172.21.15.64:0/1229331572 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 -- 172.21.15.64:0/1229331572 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 998575437 0 0) 0x7f2d6809dcb0 con 0x7f2d68135f70 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient(hunting): my global_id is 114207 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 -- 172.21.15.64:0/1229331572 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d54003520 con 0x7f2d68135f70 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 -- 172.21.15.64:0/1229331572 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2499272473 0 0) 0x7f2d6809d6f0 con 0x7f2d68135b50 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient(hunting): my global_id is 4943 2024-09-17T15:48:05.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 -- 172.21.15.64:0/1229331572 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d6809dcb0 con 0x7f2d68135b50 2024-09-17T15:48:05.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 -- 172.21.15.64:0/1229331572 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3405371564 0 0) 0x7f2d6809d9d0 con 0x7f2d680a5520 2024-09-17T15:48:05.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient(hunting): my global_id is 114195 2024-09-17T15:48:05.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 -- 172.21.15.64:0/1229331572 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d6809d6f0 con 0x7f2d680a5520 2024-09-17T15:48:05.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 -- 172.21.15.64:0/1229331572 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3328658923 0 0) 0x7f2d6809dcb0 con 0x7f2d68135b50 2024-09-17T15:48:05.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 -- 172.21.15.64:0/1229331572 >> v1:172.21.15.64:6792/0 conn(0x7f2d68135f70 legacy=0x7f2d681415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 -- 172.21.15.64:0/1229331572 >> v1:172.21.15.100:6790/0 conn(0x7f2d680a5520 legacy=0x7f2d680a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient: _finish_hunting 0 2024-09-17T15:48:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 monclient: found mon.noname-b 2024-09-17T15:48:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:48:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 -- 172.21.15.64:0/1229331572 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d6809ccd0 con 0x7f2d68135b50 2024-09-17T15:48:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient: _finish_auth 0 2024-09-17T15:48:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient: _check_auth_tickets 2024-09-17T15:48:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 -- 172.21.15.64:0/1229331572 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2d58001810 con 0x7f2d68135b50 2024-09-17T15:48:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6fad6640 5 monclient: authenticate success, global_id 4943 2024-09-17T15:48:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6fad6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:05.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient: dump: 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.677 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6fad6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 -- 172.21.15.64:0/1229331572 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d580019b0 con 0x7f2d68135b50 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 1 -- 172.21.15.64:0/1229331572 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2d58004b20 con 0x7f2d68135b50 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6c870640 10 monclient: dump: 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6fad6640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.662+0000 7f2d6fad6640 4 set_mon_vals no callback set 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.663+0000 7f2d6fad6640 10 monclient: shutdown 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.663+0000 7f2d6fad6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.663+0000 7f2d6fad6640 1 -- 172.21.15.64:0/1229331572 >> v1:172.21.15.64:6790/0 conn(0x7f2d68135b50 legacy=0x7f2d6813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.663+0000 7f2d6fad6640 1 -- 172.21.15.64:0/1229331572 shutdown_connections 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.663+0000 7f2d6fad6640 1 -- 172.21.15.64:0/1229331572 >> 172.21.15.64:0/1229331572 conn(0x7f2d6806db30 msgr2=0x7f2d680ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.663+0000 7f2d6fad6640 1 -- 172.21.15.64:0/1229331572 shutdown_connections 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.663+0000 7f2d6fad6640 1 -- 172.21.15.64:0/1229331572 wait complete. 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 10 monclient: build_initial_monmap 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 10 monclient: monmap: 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:05.680 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 1 Processor -- start 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 1 -- start start 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 10 monclient: init 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 10 monclient: _add_conns ranks=[6,7,3,2,4,5,1,8,0] 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 10 monclient(hunting): picked mon.noname-d con 0x7f2d680a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 10 monclient(hunting): picked mon.noname-i con 0x7f2d68135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:05.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 10 monclient(hunting): picked mon.noname-g con 0x7f2d68135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:05.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d68138810 con 0x7f2d68135f70 2024-09-17T15:48:05.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d680a6760 con 0x7f2d680a5520 2024-09-17T15:48:05.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d68135270 con 0x7f2d68135b50 2024-09-17T15:48:05.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:05.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6fad6640 10 monclient(hunting): authenticate will time out at 883.340943s 2024-09-17T15:48:05.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6d872640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f2d680a5520 0x7f2d68136e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:50320/0 (socket says 172.21.15.64:50320) 2024-09-17T15:48:05.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.664+0000 7f2d6d872640 1 -- 172.21.15.64:0/41355988 learned_addr learned my addr 172.21.15.64:0/41355988 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:05.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 594366233 0 0) 0x7f2d680a6760 con 0x7f2d680a5520 2024-09-17T15:48:05.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): my global_id is 114216 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d4c003520 con 0x7f2d680a5520 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3337605204 0 0) 0x7f2d68138810 con 0x7f2d68135f70 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): my global_id is 114204 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d680a6760 con 0x7f2d68135f70 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2813031438 0 0) 0x7f2d4c003520 con 0x7f2d680a5520 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d68138810 con 0x7f2d680a5520 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2d64001bb0 con 0x7f2d680a5520 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): dump: 2024-09-17T15:48:05.683 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.684 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.684 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.684 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.684 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.684 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.684 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.684 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.684 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.684 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.684 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.684 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.684 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.684 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.685 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.685 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:05.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 847395823 0 0) 0x7f2d68135270 con 0x7f2d68135b50 2024-09-17T15:48:05.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient(hunting): my global_id is 5065 2024-09-17T15:48:05.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d4c003520 con 0x7f2d68135b50 2024-09-17T15:48:05.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2674663897 0 0) 0x7f2d68138810 con 0x7f2d680a5520 2024-09-17T15:48:05.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 >> v1:172.21.15.100:6792/0 conn(0x7f2d68135b50 legacy=0x7f2d68137580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 >> v1:172.21.15.100:6790/0 conn(0x7f2d68135f70 legacy=0x7f2d681d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:05.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 monclient: found mon.d 2024-09-17T15:48:05.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:05.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:48:05.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d68137c90 con 0x7f2d680a5520 2024-09-17T15:48:05.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:48:05.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:48:05.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:05.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d6fad6640 5 monclient: authenticate success, global_id 114216 2024-09-17T15:48:05.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d6fad6640 10 monclient: _renew_subs 2024-09-17T15:48:05.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d6fad6640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:48:05.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d6fad6640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2d6813ab70 con 0x7f2d680a5520 2024-09-17T15:48:05.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d6fad6640 10 monclient: _renew_subs 2024-09-17T15:48:05.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d6fad6640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:48:05.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d6fad6640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f2d68139370 con 0x7f2d680a5520 2024-09-17T15:48:05.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d640038c0 con 0x7f2d680a5520 2024-09-17T15:48:05.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:05.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2d64003e30 con 0x7f2d680a5520 2024-09-17T15:48:05.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:48:05.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5ffff640 4 set_mon_vals no callback set 2024-09-17T15:48:05.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.665+0000 7f2d5e7fc640 10 monclient: dump: 2024-09-17T15:48:05.687 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.687 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.688 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.688 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.688 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.688 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.688 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.688 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.688 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.688 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.688 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.688 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.688 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.688 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.688 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.689 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.666+0000 7f2d6fad6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:05.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.666+0000 7f2d6fad6640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2d500065c0 con 0x7f2d50002860 2024-09-17T15:48:05.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.666+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f2d64003fd0 con 0x7f2d680a5520 2024-09-17T15:48:05.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(111..111 src has 1..111) ==== 3898+0+0 (unknown 2549879297 0 0) 0x7f2d6404eb80 con 0x7f2d680a5520 2024-09-17T15:48:05.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 648388144 0 0) 0x7f2d500065c0 con 0x7f2d50002860 2024-09-17T15:48:05.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 10 monclient(hunting): my global_id is 4952 2024-09-17T15:48:05.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d68138810 con 0x7f2d50002860 2024-09-17T15:48:05.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d6e073640 10 monclient: get_auth_request con 0x7f2d4c040310 auth_method 0 2024-09-17T15:48:05.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3985613238 0 0) 0x7f2d68138810 con 0x7f2d50002860 2024-09-17T15:48:05.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d500065c0 con 0x7f2d50002860 2024-09-17T15:48:05.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2d580017d0 con 0x7f2d50002860 2024-09-17T15:48:05.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:05.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 10 monclient: dump: 2024-09-17T15:48:05.690 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.690 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.690 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.690 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 156573488 0 0) 0x7f2d500065c0 con 0x7f2d50002860 2024-09-17T15:48:05.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.64:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d5000a160 con 0x7f2d50002860 2024-09-17T15:48:05.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2d580049d0 con 0x7f2d50002860 2024-09-17T15:48:05.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:05.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:05.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 >> v1:172.21.15.64:6790/0 conn(0x7f2d50002860 legacy=0x7f2d50004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.667+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 >> v1:172.21.15.64:6790/0 conn(0x7f2d50002860 legacy=0x7f2d50004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:05.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.673+0000 7f2d6fad6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:05.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.673+0000 7f2d6fad6640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2d50009e70 con 0x7f2d50009130 2024-09-17T15:48:05.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1893623207 0 0) 0x7f2d50009e70 con 0x7f2d50009130 2024-09-17T15:48:05.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 10 monclient(hunting): my global_id is 4961 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d500065c0 con 0x7f2d50009130 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2013243588 0 0) 0x7f2d500065c0 con 0x7f2d50009130 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d50009e70 con 0x7f2d50009130 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2d58005000 con 0x7f2d50009130 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 10 monclient: dump: 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.694 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 980040312 0 0) 0x7f2d50009e70 con 0x7f2d50009130 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 --> v1:172.21.15.64:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d5000c410 con 0x7f2d50009130 2024-09-17T15:48:05.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 3641860299) 0x7f2d580064d0 con 0x7f2d50009130 2024-09-17T15:48:05.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:05.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:05.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 >> v1:172.21.15.64:6790/0 conn(0x7f2d50009130 legacy=0x7f2d500095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.674+0000 7f2d5e7fc640 1 -- 172.21.15.64:0/41355988 >> v1:172.21.15.64:6790/0 conn(0x7f2d50009130 legacy=0x7f2d500095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:05.696 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:05.696 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 142319, 2024-09-17T15:48:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.697 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 3, 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:05.698 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:05.699 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:05.700 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:05.701 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:05.702 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.703 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.703 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.703 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:05.703 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.703 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.703 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.703 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:05.703 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:05.703 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.703 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.704 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.704 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:05.704 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:05.704 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.704 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.704 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.704 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:05.704 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.704 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.705 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.706 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.706 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:05.706 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.706 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.706 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.706 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:05.706 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:05.707 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.707 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.707 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.707 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:05.708 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:05.708 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.708 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.708 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.708 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:05.709 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.709 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.709 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:05.709 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:05.710 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.710 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.710 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.710 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:05.710 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:05.710 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.711 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.711 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.711 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:05.711 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.711 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.711 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:05.711 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:05.712 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.712 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.712 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.712 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:05.712 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:05.712 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.712 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.712 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.713 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:05.713 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.713 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.713 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:05.713 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:05.713 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.713 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.713 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.713 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.714 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.715 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:05.716 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:05.716 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.716 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.716 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.716 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:05.716 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:05.716 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.716 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.717 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.717 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:05.717 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.717 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.717 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:05.717 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:05.717 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.717 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.717 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.717 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:05.718 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:05.719 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.719 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.719 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:05.719 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:05.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.675+0000 7f2d6fad6640 1 -- 172.21.15.64:0/41355988 >> v1:172.21.15.100:6812/4210963484 conn(0x7f2d4c040310 legacy=0x7f2d4c0427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.675+0000 7f2d6fad6640 10 monclient: shutdown 2024-09-17T15:48:05.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.675+0000 7f2d6fad6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:05.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.675+0000 7f2d6fad6640 1 -- 172.21.15.64:0/41355988 >> v1:172.21.15.64:6792/0 conn(0x7f2d680a5520 legacy=0x7f2d68136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.675+0000 7f2d6e073640 1 -- 172.21.15.64:0/41355988 reap_dead start 2024-09-17T15:48:05.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.675+0000 7f2d6e073640 1 -- 172.21.15.64:0/41355988 reap_dead start 2024-09-17T15:48:05.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.675+0000 7f2d6fad6640 1 -- 172.21.15.64:0/41355988 shutdown_connections 2024-09-17T15:48:05.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.675+0000 7f2d6fad6640 1 -- 172.21.15.64:0/41355988 >> 172.21.15.64:0/41355988 conn(0x7f2d6806db30 msgr2=0x7f2d680a4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:05.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.675+0000 7f2d6fad6640 1 -- 172.21.15.64:0/41355988 shutdown_connections 2024-09-17T15:48:05.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.675+0000 7f2d6fad6640 1 -- 172.21.15.64:0/41355988 wait complete. 2024-09-17T15:48:05.722 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:05.722 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-17T15:48:05.722 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:48:05.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.822+0000 7fee4ccef640 10 monclient: get_monmap_and_config 2024-09-17T15:48:05.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.822+0000 7fee4ccef640 10 monclient: build_initial_monmap 2024-09-17T15:48:05.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.822+0000 7fee4ccef640 10 monclient: monmap: 2024-09-17T15:48:05.825 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:05.825 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.825 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:05.824313+0000 2024-09-17T15:48:05.825 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:05.824313+0000 2024-09-17T15:48:05.825 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:05.825 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.825 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:05.826 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:05.826 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:05.826 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:05.826 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:05.826 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:05.826 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:05.826 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:05.826 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:05.826 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4ccef640 1 Processor -- start 2024-09-17T15:48:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4ccef640 1 -- start start 2024-09-17T15:48:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4ccef640 10 monclient: init 2024-09-17T15:48:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4ccef640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:05.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4ccef640 10 monclient: _add_conns ranks=[4,8,5,0,2,6,3,7,1] 2024-09-17T15:48:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4ccef640 10 monclient(hunting): picked mon.noname-c con 0x7fee480a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4ccef640 10 monclient(hunting): picked mon.noname-e con 0x7fee48135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4ccef640 10 monclient(hunting): picked mon.noname-h con 0x7fee48135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4ccef640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee4809d6f0 con 0x7fee480a5520 2024-09-17T15:48:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4ccef640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee4809d9d0 con 0x7fee48135f70 2024-09-17T15:48:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4ccef640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee4809dcb0 con 0x7fee48135b50 2024-09-17T15:48:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4ccef640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4ccef640 10 monclient(hunting): authenticate will time out at 883.499884s 2024-09-17T15:48:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4659c640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fee480a5520 0x7fee480a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:58814/0 (socket says 172.21.15.64:58814) 2024-09-17T15:48:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.823+0000 7fee4659c640 1 -- 172.21.15.64:0/43832421 learned_addr learned my addr 172.21.15.64:0/43832421 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:05.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee4659c640 1 -- 172.21.15.64:0/43832421 >> v1:172.21.15.64:6791/0 conn(0x7fee480a5520 legacy=0x7fee480a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-17T15:48:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee4659c640 1 -- 172.21.15.64:0/43832421 >> v1:172.21.15.64:6791/0 conn(0x7fee480a5520 legacy=0x7fee480a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:48:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee4659c640 1 --1- 172.21.15.64:0/43832421 >> v1:172.21.15.64:6791/0 conn(0x7fee480a5520 0x7fee480a5900 :-1 s=OPENED pgs=210 cs=1 l=1).handle_message read tag failed 2024-09-17T15:48:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee4659c640 1 --1- 172.21.15.64:0/43832421 >> v1:172.21.15.64:6791/0 conn(0x7fee480a5520 0x7fee480a5900 :-1 s=OPENED pgs=210 cs=1 l=1).fault on lossy channel, failing 2024-09-17T15:48:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.64:6791/0 2024-09-17T15:48:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 1 -- 172.21.15.64:0/43832421 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3418516000 0 0) 0x7fee4809dcb0 con 0x7fee48135b50 2024-09-17T15:48:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient(hunting): my global_id is 5120 2024-09-17T15:48:05.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 1 -- 172.21.15.64:0/43832421 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee34003520 con 0x7fee48135b50 2024-09-17T15:48:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 1 -- 172.21.15.64:0/43832421 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3735802164 0 0) 0x7fee4809d9d0 con 0x7fee48135f70 2024-09-17T15:48:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient(hunting): my global_id is 114233 2024-09-17T15:48:05.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 1 -- 172.21.15.64:0/43832421 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee4809dcb0 con 0x7fee48135f70 2024-09-17T15:48:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 1 -- 172.21.15.64:0/43832421 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3544940733 0 0) 0x7fee34003520 con 0x7fee48135b50 2024-09-17T15:48:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 1 -- 172.21.15.64:0/43832421 >> v1:172.21.15.100:6791/0 conn(0x7fee48135f70 legacy=0x7fee481415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 1 -- 172.21.15.64:0/43832421 >> v1:172.21.15.64:6791/0 conn(0x7fee480a5520 legacy=0x7fee480a5900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:48:05.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 1 monclient: found mon.noname-e 2024-09-17T15:48:05.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:05.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:48:05.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 1 -- 172.21.15.64:0/43832421 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4809ccd0 con 0x7fee48135b50 2024-09-17T15:48:05.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient: _finish_auth 0 2024-09-17T15:48:05.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:48:05.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 1 -- 172.21.15.64:0/43832421 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fee30001830 con 0x7fee48135b50 2024-09-17T15:48:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee4ccef640 5 monclient: authenticate success, global_id 5120 2024-09-17T15:48:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee4ccef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:48:05.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient: dump: 2024-09-17T15:48:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.842 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.843 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.844 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee4ccef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:05.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 1 -- 172.21.15.64:0/43832421 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee30001a50 con 0x7fee48135b50 2024-09-17T15:48:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 1 -- 172.21.15.64:0/43832421 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fee30003950 con 0x7fee48135b50 2024-09-17T15:48:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee4ccef640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee4ccef640 4 set_mon_vals no callback set 2024-09-17T15:48:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.824+0000 7fee45d9b640 10 monclient: dump: 2024-09-17T15:48:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.846 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.847 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.848 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.848 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.848 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.848 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.848 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.825+0000 7fee4ccef640 10 monclient: shutdown 2024-09-17T15:48:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.825+0000 7fee4ccef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.825+0000 7fee4ccef640 1 -- 172.21.15.64:0/43832421 >> v1:172.21.15.64:6793/0 conn(0x7fee48135b50 legacy=0x7fee4813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.825+0000 7fee4ccef640 1 -- 172.21.15.64:0/43832421 shutdown_connections 2024-09-17T15:48:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.825+0000 7fee4ccef640 1 -- 172.21.15.64:0/43832421 >> 172.21.15.64:0/43832421 conn(0x7fee4806db30 msgr2=0x7fee480ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:05.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.825+0000 7fee4ccef640 1 -- 172.21.15.64:0/43832421 shutdown_connections 2024-09-17T15:48:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.825+0000 7fee4ccef640 1 -- 172.21.15.64:0/43832421 wait complete. 2024-09-17T15:48:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.825+0000 7fee4ccef640 10 monclient: build_initial_monmap 2024-09-17T15:48:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.825+0000 7fee4ccef640 10 monclient: monmap: 2024-09-17T15:48:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:05.851 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:05.851 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:05.851 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:05.851 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:05.851 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:05.851 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:05.852 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:05.852 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:05.852 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:05.852 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:05.852 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:05.852 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:05.852 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4ccef640 1 Processor -- start 2024-09-17T15:48:05.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4ccef640 1 -- start start 2024-09-17T15:48:05.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4ccef640 10 monclient: init 2024-09-17T15:48:05.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4ccef640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:05.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4ccef640 10 monclient: _add_conns ranks=[8,6,4,2,5,3,0,7,1] 2024-09-17T15:48:05.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4ccef640 10 monclient(hunting): picked mon.noname-e con 0x7fee480a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:05.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4ccef640 10 monclient(hunting): picked mon.noname-d con 0x7fee48135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:05.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4ccef640 10 monclient(hunting): picked mon.noname-c con 0x7fee48135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:05.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4ccef640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee48135350 con 0x7fee48135f70 2024-09-17T15:48:05.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4ccef640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee480a9300 con 0x7fee48135b50 2024-09-17T15:48:05.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4ccef640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee480a0610 con 0x7fee480a5520 2024-09-17T15:48:05.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4ccef640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:05.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4ccef640 10 monclient(hunting): authenticate will time out at 883.502827s 2024-09-17T15:48:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4659c640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fee480a5520 0x7fee48132050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38698/0 (socket says 172.21.15.64:38698) 2024-09-17T15:48:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.826+0000 7fee4659c640 1 -- 172.21.15.64:0/3896108143 learned_addr learned my addr 172.21.15.64:0/3896108143 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3755267956 0 0) 0x7fee48135350 con 0x7fee48135f70 2024-09-17T15:48:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): my global_id is 5170 2024-09-17T15:48:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee28003550 con 0x7fee48135f70 2024-09-17T15:48:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2135303032 0 0) 0x7fee480a0610 con 0x7fee480a5520 2024-09-17T15:48:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): my global_id is 5129 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee48135350 con 0x7fee480a5520 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3249360088 0 0) 0x7fee480a9300 con 0x7fee48135b50 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): my global_id is 114225 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee480a0610 con 0x7fee48135b50 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3802217071 0 0) 0x7fee28003550 con 0x7fee48135f70 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee480a9300 con 0x7fee48135f70 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fee40001c70 con 0x7fee48135f70 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): dump: 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1627237425 0 0) 0x7fee48135350 con 0x7fee480a5520 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee28003550 con 0x7fee480a5520 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4084657871 0 0) 0x7fee480a0610 con 0x7fee48135b50 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee48135350 con 0x7fee48135b50 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fee38001b10 con 0x7fee480a5520 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): dump: 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.859 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fee300017f0 con 0x7fee48135b50 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient(hunting): dump: 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3918888122 0 0) 0x7fee480a9300 con 0x7fee48135f70 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 >> v1:172.21.15.64:6793/0 conn(0x7fee480a5520 legacy=0x7fee48132050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 >> v1:172.21.15.64:6792/0 conn(0x7fee48135b50 legacy=0x7fee48132760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient: _finish_hunting 0 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 monclient: found mon.c 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee480a08f0 con 0x7fee48135f70 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient: _finish_auth 0 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee4ccef640 5 monclient: authenticate success, global_id 5170 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee4ccef640 10 monclient: _renew_subs 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee4ccef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee4ccef640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fee4809f310 con 0x7fee48135f70 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee4ccef640 10 monclient: _renew_subs 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee4ccef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee4ccef640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fee48133050 con 0x7fee48135f70 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee40003990 con 0x7fee48135f70 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fee40003f00 con 0x7fee48135f70 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee4559a640 4 set_mon_vals no callback set 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.827+0000 7fee3effd640 10 monclient: dump: 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.864 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.828+0000 7fee4ccef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.828+0000 7fee4ccef640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fee2c0065c0 con 0x7fee2c002860 2024-09-17T15:48:05.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.828+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fee400040a0 con 0x7fee48135f70 2024-09-17T15:48:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(111..111 src has 1..111) ==== 3898+0+0 (unknown 2549879297 0 0) 0x7fee4004dbe0 con 0x7fee48135f70 2024-09-17T15:48:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2917877504 0 0) 0x7fee2c0065c0 con 0x7fee2c002860 2024-09-17T15:48:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 10 monclient(hunting): my global_id is 5179 2024-09-17T15:48:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee480a9300 con 0x7fee2c002860 2024-09-17T15:48:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3ffff640 10 monclient: get_auth_request con 0x7fee28040de0 auth_method 0 2024-09-17T15:48:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3103634536 0 0) 0x7fee480a9300 con 0x7fee2c002860 2024-09-17T15:48:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee2c0065c0 con 0x7fee2c002860 2024-09-17T15:48:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fee38004b60 con 0x7fee2c002860 2024-09-17T15:48:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 10 monclient: dump: 2024-09-17T15:48:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.867 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2595151636 0 0) 0x7fee2c0065c0 con 0x7fee2c002860 2024-09-17T15:48:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee2c00a160 con 0x7fee2c002860 2024-09-17T15:48:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fee380058d0 con 0x7fee2c002860 2024-09-17T15:48:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 >> v1:172.21.15.64:6791/0 conn(0x7fee2c002860 legacy=0x7fee2c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.829+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 >> v1:172.21.15.64:6791/0 conn(0x7fee2c002860 legacy=0x7fee2c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.835+0000 7fee4ccef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.835+0000 7fee4ccef640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fee2c009e50 con 0x7fee2c009110 2024-09-17T15:48:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 445795949 0 0) 0x7fee2c009e50 con 0x7fee2c009110 2024-09-17T15:48:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 10 monclient(hunting): my global_id is 5188 2024-09-17T15:48:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee2c0065c0 con 0x7fee2c009110 2024-09-17T15:48:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2500643334 0 0) 0x7fee2c0065c0 con 0x7fee2c009110 2024-09-17T15:48:05.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee2c009e50 con 0x7fee2c009110 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fee380055c0 con 0x7fee2c009110 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 10 monclient: dump: 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:05.870 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:05.871 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 511708715 0 0) 0x7fee2c009e50 con 0x7fee2c009110 2024-09-17T15:48:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 --> v1:172.21.15.64:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fee2c00c3f0 con 0x7fee2c009110 2024-09-17T15:48:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3308172650) 0x7fee38005b00 con 0x7fee2c009110 2024-09-17T15:48:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 >> v1:172.21.15.64:6791/0 conn(0x7fee2c009110 legacy=0x7fee2c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.836+0000 7fee3effd640 1 -- 172.21.15.64:0/3896108143 >> v1:172.21.15.64:6791/0 conn(0x7fee2c009110 legacy=0x7fee2c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:05.872 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 142494, 2024-09-17T15:48:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:48:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:05.899 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:05.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.837+0000 7fee4ccef640 1 -- 172.21.15.64:0/3896108143 >> v1:172.21.15.100:6812/4210963484 conn(0x7fee28040de0 legacy=0x7fee280432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.837+0000 7fee4ccef640 10 monclient: shutdown 2024-09-17T15:48:05.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.837+0000 7fee4ccef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:05.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.837+0000 7fee4ccef640 1 -- 172.21.15.64:0/3896108143 >> v1:172.21.15.64:6791/0 conn(0x7fee48135f70 legacy=0x7fee481d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:05.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.837+0000 7fee46d9d640 1 -- 172.21.15.64:0/3896108143 reap_dead start 2024-09-17T15:48:05.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.837+0000 7fee46d9d640 1 -- 172.21.15.64:0/3896108143 reap_dead start 2024-09-17T15:48:05.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.837+0000 7fee4ccef640 1 -- 172.21.15.64:0/3896108143 shutdown_connections 2024-09-17T15:48:05.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.837+0000 7fee4ccef640 1 -- 172.21.15.64:0/3896108143 >> 172.21.15.64:0/3896108143 conn(0x7fee4806db30 msgr2=0x7fee4812c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:05.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.837+0000 7fee4ccef640 1 -- 172.21.15.64:0/3896108143 shutdown_connections 2024-09-17T15:48:05.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:05.837+0000 7fee4ccef640 1 -- 172.21.15.64:0/3896108143 wait complete. 2024-09-17T15:48:05.901 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:05.901 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-17T15:48:05.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-17T15:48:05.972 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:05.972 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36565]: starting. 2024-09-17T15:48:05.972 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36565]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36412 2024-09-17T15:48:05.972 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36565]: created object 0... 2024-09-17T15:48:05.972 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36565]: created object 25... 2024-09-17T15:48:05.973 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36565]: created object 49... 2024-09-17T15:48:05.973 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36565]: finishing. 2024-09-17T15:48:05.973 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36565]: shutting down. 2024-09-17T15:48:05.982 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:05.983 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36567]: starting. 2024-09-17T15:48:05.983 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36567]: listing objects. 2024-09-17T15:48:05.983 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36567]: listed object 0... 2024-09-17T15:48:05.983 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36567]: listed object 25... 2024-09-17T15:48:05.983 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36567]: saw 50 objects 2024-09-17T15:48:05.983 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36567]: shutting down. 2024-09-17T15:48:06.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.010+0000 7f2435597640 10 monclient: get_monmap_and_config 2024-09-17T15:48:06.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.010+0000 7f2435597640 10 monclient: build_initial_monmap 2024-09-17T15:48:06.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.010+0000 7f2435597640 10 monclient: monmap: 2024-09-17T15:48:06.023 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:06.023 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.023 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:06.012581+0000 2024-09-17T15:48:06.023 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:06.012581+0000 2024-09-17T15:48:06.024 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:06.024 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.024 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:06.024 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:06.024 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:06.024 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:06.024 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:06.024 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:06.024 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:06.024 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:06.024 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:06.025 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.011+0000 7f2435597640 1 Processor -- start 2024-09-17T15:48:06.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.011+0000 7f2435597640 1 -- start start 2024-09-17T15:48:06.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.011+0000 7f2435597640 10 monclient: init 2024-09-17T15:48:06.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.011+0000 7f2435597640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:06.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.011+0000 7f2435597640 10 monclient: _add_conns ranks=[8,4,5,1,2,0,3,7,6] 2024-09-17T15:48:06.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.011+0000 7f2435597640 10 monclient(hunting): picked mon.noname-e con 0x7f24300a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:06.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.011+0000 7f2435597640 10 monclient(hunting): picked mon.noname-c con 0x7f2430135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:06.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.011+0000 7f2435597640 10 monclient(hunting): picked mon.noname-h con 0x7f2430135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:06.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.011+0000 7f2435597640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f243009d6f0 con 0x7f2430135b50 2024-09-17T15:48:06.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.011+0000 7f2435597640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f243009d9d0 con 0x7f2430135f70 2024-09-17T15:48:06.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f2435597640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f243009dcb0 con 0x7f24300a5520 2024-09-17T15:48:06.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f2435597640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:06.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f2435597640 10 monclient(hunting): authenticate will time out at 883.688055s 2024-09-17T15:48:06.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242e7fc640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f2430135b50 0x7f243013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:58850/0 (socket says 172.21.15.64:58850) 2024-09-17T15:48:06.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242e7fc640 1 -- 172.21.15.64:0/1785314124 learned_addr learned my addr 172.21.15.64:0/1785314124 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:06.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242effd640 1 --1- 172.21.15.64:0/1785314124 >> v1:172.21.15.64:6793/0 conn(0x7f24300a5520 0x7f24300a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38704/0 (socket says 172.21.15.64:38704) 2024-09-17T15:48:06.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 -- 172.21.15.64:0/1785314124 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2501684459 0 0) 0x7f243009dcb0 con 0x7f24300a5520 2024-09-17T15:48:06.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient(hunting): my global_id is 5138 2024-09-17T15:48:06.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 -- 172.21.15.64:0/1785314124 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2418003520 con 0x7f24300a5520 2024-09-17T15:48:06.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 -- 172.21.15.64:0/1785314124 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3200158974 0 0) 0x7f243009d6f0 con 0x7f2430135b50 2024-09-17T15:48:06.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient(hunting): my global_id is 5197 2024-09-17T15:48:06.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 -- 172.21.15.64:0/1785314124 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f243009dcb0 con 0x7f2430135b50 2024-09-17T15:48:06.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 -- 172.21.15.64:0/1785314124 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 697902171 0 0) 0x7f243009d9d0 con 0x7f2430135f70 2024-09-17T15:48:06.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient(hunting): my global_id is 114242 2024-09-17T15:48:06.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 -- 172.21.15.64:0/1785314124 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f243009d6f0 con 0x7f2430135f70 2024-09-17T15:48:06.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 -- 172.21.15.64:0/1785314124 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2053130996 0 0) 0x7f243009dcb0 con 0x7f2430135b50 2024-09-17T15:48:06.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 -- 172.21.15.64:0/1785314124 >> v1:172.21.15.64:6793/0 conn(0x7f24300a5520 legacy=0x7f24300a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 -- 172.21.15.64:0/1785314124 >> v1:172.21.15.100:6791/0 conn(0x7f2430135f70 legacy=0x7f24301415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient: _finish_hunting 0 2024-09-17T15:48:06.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 monclient: found mon.noname-c 2024-09-17T15:48:06.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:06.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:48:06.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 -- 172.21.15.64:0/1785314124 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f243009ccd0 con 0x7f2430135b50 2024-09-17T15:48:06.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient: _finish_auth 0 2024-09-17T15:48:06.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:48:06.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:06.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f2435597640 5 monclient: authenticate success, global_id 5197 2024-09-17T15:48:06.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f2435597640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:06.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 -- 172.21.15.64:0/1785314124 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2424001810 con 0x7f2430135b50 2024-09-17T15:48:06.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:06.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient: dump: 2024-09-17T15:48:06.029 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.029 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.030 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.030 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.030 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.030 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.030 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.030 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.030 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.030 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.030 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.030 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.030 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.031 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.031 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.031 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f2435597640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:06.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 -- 172.21.15.64:0/1785314124 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2424001a20 con 0x7f2430135b50 2024-09-17T15:48:06.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:06.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 1 -- 172.21.15.64:0/1785314124 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2424004be0 con 0x7f2430135b50 2024-09-17T15:48:06.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f2435597640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:06.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f2435597640 4 set_mon_vals no callback set 2024-09-17T15:48:06.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:06.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.012+0000 7f242dffb640 10 monclient: dump: 2024-09-17T15:48:06.032 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.032 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.032 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.032 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.032 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.032 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.032 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.032 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.032 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.032 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.032 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.033 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.033 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.033 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.033 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.033 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.013+0000 7f2435597640 10 monclient: shutdown 2024-09-17T15:48:06.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.013+0000 7f2435597640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:06.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.013+0000 7f2435597640 1 -- 172.21.15.64:0/1785314124 >> v1:172.21.15.64:6791/0 conn(0x7f2430135b50 legacy=0x7f243013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.013+0000 7f2435597640 1 -- 172.21.15.64:0/1785314124 shutdown_connections 2024-09-17T15:48:06.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.013+0000 7f2435597640 1 -- 172.21.15.64:0/1785314124 >> 172.21.15.64:0/1785314124 conn(0x7f243006db30 msgr2=0x7f24300ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:06.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.013+0000 7f2435597640 1 -- 172.21.15.64:0/1785314124 shutdown_connections 2024-09-17T15:48:06.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.013+0000 7f2435597640 1 -- 172.21.15.64:0/1785314124 wait complete. 2024-09-17T15:48:06.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.013+0000 7f2435597640 10 monclient: build_initial_monmap 2024-09-17T15:48:06.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.013+0000 7f2435597640 10 monclient: monmap: 2024-09-17T15:48:06.034 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:06.034 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:06.034 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:06.034 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:06.034 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:06.034 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:06.034 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:06.034 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:06.035 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:06.035 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:06.035 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:06.035 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:06.035 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:06.035 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:06.035 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:06.035 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f2435597640 1 Processor -- start 2024-09-17T15:48:06.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f2435597640 1 -- start start 2024-09-17T15:48:06.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f2435597640 10 monclient: init 2024-09-17T15:48:06.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f2435597640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:06.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f2435597640 10 monclient: _add_conns ranks=[1,3,8,0,5,6,2,7,4] 2024-09-17T15:48:06.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f2435597640 10 monclient(hunting): picked mon.noname-f con 0x7f24300a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:06.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f2435597640 10 monclient(hunting): picked mon.noname-g con 0x7f2430135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:06.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f2435597640 10 monclient(hunting): picked mon.noname-e con 0x7f2430135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:06.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f2435597640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f243013a510 con 0x7f24300a5520 2024-09-17T15:48:06.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f2435597640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24300a6760 con 0x7f2430135b50 2024-09-17T15:48:06.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f2435597640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2430138a50 con 0x7f2430135f70 2024-09-17T15:48:06.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f2435597640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:06.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f2435597640 10 monclient(hunting): authenticate will time out at 883.690889s 2024-09-17T15:48:06.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f242f7fe640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f2430135f70 0x7f24301d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38708/0 (socket says 172.21.15.64:38708) 2024-09-17T15:48:06.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.014+0000 7f242f7fe640 1 -- 172.21.15.64:0/2804375263 learned_addr learned my addr 172.21.15.64:0/2804375263 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:06.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1083905759 0 0) 0x7f2430138a50 con 0x7f2430135f70 2024-09-17T15:48:06.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): my global_id is 5147 2024-09-17T15:48:06.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f241c003520 con 0x7f2430135f70 2024-09-17T15:48:06.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1584500811 0 0) 0x7f241c003520 con 0x7f2430135f70 2024-09-17T15:48:06.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2430138a50 con 0x7f2430135f70 2024-09-17T15:48:06.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2428001a90 con 0x7f2430135f70 2024-09-17T15:48:06.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:48:06.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): dump: 2024-09-17T15:48:06.038 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.038 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.038 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.038 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.038 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.038 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.038 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.038 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.038 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.038 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.039 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.039 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.039 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.039 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.039 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.039 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:06.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3774803438 0 0) 0x7f24300a6760 con 0x7f2430135b50 2024-09-17T15:48:06.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): my global_id is 114213 2024-09-17T15:48:06.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f241c003520 con 0x7f2430135b50 2024-09-17T15:48:06.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2753866498 0 0) 0x7f2420001e30 con 0x7f24300a5520 2024-09-17T15:48:06.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient(hunting): my global_id is 5230 2024-09-17T15:48:06.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24300a6760 con 0x7f24300a5520 2024-09-17T15:48:06.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3415514358 0 0) 0x7f2430138a50 con 0x7f2430135f70 2024-09-17T15:48:06.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 >> v1:172.21.15.100:6790/0 conn(0x7f2430135b50 legacy=0x7f2430137670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 >> v1:172.21.15.100:6789/0 conn(0x7f24300a5520 legacy=0x7f2430136f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient: _finish_hunting 0 2024-09-17T15:48:06.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 monclient: found mon.e 2024-09-17T15:48:06.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:06.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:06.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2430137d80 con 0x7f2430135f70 2024-09-17T15:48:06.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient: _finish_auth 0 2024-09-17T15:48:06.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient: _check_auth_tickets 2024-09-17T15:48:06.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:06.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f2435597640 5 monclient: authenticate success, global_id 5147 2024-09-17T15:48:06.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f2435597640 10 monclient: _renew_subs 2024-09-17T15:48:06.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f2435597640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:06.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f2435597640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f243013adb0 con 0x7f2430135f70 2024-09-17T15:48:06.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f2435597640 10 monclient: _renew_subs 2024-09-17T15:48:06.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f2435597640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:06.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f2435597640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f2430139820 con 0x7f2430135f70 2024-09-17T15:48:06.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f24280038c0 con 0x7f2430135f70 2024-09-17T15:48:06.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:06.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2428003e70 con 0x7f2430135f70 2024-09-17T15:48:06.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f242d7fa640 4 set_mon_vals no callback set 2024-09-17T15:48:06.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:06.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.015+0000 7f24177fe640 10 monclient: dump: 2024-09-17T15:48:06.043 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.043 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.043 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.043 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.043 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.043 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.043 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.043 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.043 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.043 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.044 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.044 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.044 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.044 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.044 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.044 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.016+0000 7f2435597640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:06.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.016+0000 7f2435597640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f24100065c0 con 0x7f2410002860 2024-09-17T15:48:06.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.016+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f2428004010 con 0x7f2430135f70 2024-09-17T15:48:06.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.016+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(112..112 src has 1..112) ==== 3898+0+0 (unknown 4054225456 0 0) 0x7f242804eb50 con 0x7f2430135f70 2024-09-17T15:48:06.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f242e7fc640 10 monclient: get_auth_request con 0x7f241c0405d0 auth_method 0 2024-09-17T15:48:06.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 326793048 0 0) 0x7f24100065c0 con 0x7f2410002860 2024-09-17T15:48:06.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 10 monclient(hunting): my global_id is 114234 2024-09-17T15:48:06.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2430138a50 con 0x7f2410002860 2024-09-17T15:48:06.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2627497249 0 0) 0x7f2430138a50 con 0x7f2410002860 2024-09-17T15:48:06.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f24100065c0 con 0x7f2410002860 2024-09-17T15:48:06.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2420001a40 con 0x7f2410002860 2024-09-17T15:48:06.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:48:06.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 10 monclient: dump: 2024-09-17T15:48:06.046 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.046 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.046 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.046 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.046 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.046 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.046 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.046 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.046 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.046 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.046 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.046 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.047 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.047 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.047 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.047 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1324229743 0 0) 0x7f24100065c0 con 0x7f2410002860 2024-09-17T15:48:06.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.64:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f241000a160 con 0x7f2410002860 2024-09-17T15:48:06.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2420004b60 con 0x7f2410002860 2024-09-17T15:48:06.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:06.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:06.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 >> v1:172.21.15.64:6792/0 conn(0x7f2410002860 legacy=0x7f2410004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.017+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 >> v1:172.21.15.64:6792/0 conn(0x7f2410002860 legacy=0x7f2410004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:06.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.023+0000 7f2435597640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:06.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.023+0000 7f2435597640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2410009e70 con 0x7f2410009130 2024-09-17T15:48:06.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 176318648 0 0) 0x7f2410009e70 con 0x7f2410009130 2024-09-17T15:48:06.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 10 monclient(hunting): my global_id is 114243 2024-09-17T15:48:06.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24100065c0 con 0x7f2410009130 2024-09-17T15:48:06.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1657656793 0 0) 0x7f24100065c0 con 0x7f2410009130 2024-09-17T15:48:06.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2410009e70 con 0x7f2410009130 2024-09-17T15:48:06.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2420005280 con 0x7f2410009130 2024-09-17T15:48:06.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:48:06.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 10 monclient: dump: 2024-09-17T15:48:06.050 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.050 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.050 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.050 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.051 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.051 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.051 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.051 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.051 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.051 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.051 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.051 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.051 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.051 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.051 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.052 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 809999796 0 0) 0x7f2410009e70 con 0x7f2410009130 2024-09-17T15:48:06.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 --> v1:172.21.15.64:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f241000c410 con 0x7f2410009130 2024-09-17T15:48:06.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 762947312) 0x7f2420007520 con 0x7f2410009130 2024-09-17T15:48:06.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:06.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:06.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 >> v1:172.21.15.64:6792/0 conn(0x7f2410009130 legacy=0x7f24100095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.024+0000 7f24177fe640 1 -- 172.21.15.64:0/2804375263 >> v1:172.21.15.64:6792/0 conn(0x7f2410009130 legacy=0x7f24100095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:06.053 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:06.053 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:06.053 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:06.053 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:06.054 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:06.054 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 9239, 2024-09-17T15:48:06.054 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:06.054 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:06.054 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:06.054 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:06.054 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:06.054 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:06.054 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:06.054 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.055 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.056 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.057 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.058 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.058 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.058 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:06.058 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:06.058 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:06.058 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:06.058 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:06.058 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:06.058 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:06.058 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:06.059 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:06.059 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:06.059 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:06.059 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:06.059 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:06.059 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:06.059 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:06.059 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:06.059 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:06.059 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.059 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.059 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.062 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.062 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:06.062 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:06.062 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.062 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.062 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.062 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:06.062 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:06.063 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.063 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.064 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:06.064 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.064 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.064 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.064 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:06.064 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:06.064 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.064 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.064 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.065 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:06.065 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:06.065 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.065 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.065 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.065 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:06.065 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.066 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.066 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:06.066 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:06.066 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.066 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.066 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.066 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:06.066 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:06.066 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.067 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.067 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:06.067 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.067 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:06.067 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:06.067 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.067 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.068 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.068 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:06.068 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:06.068 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.068 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.068 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:06.068 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.069 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:06.069 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:06.069 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.069 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.069 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.069 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:06.069 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:06.069 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.070 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.070 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.070 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:06.070 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.070 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.070 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:06.070 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:06.070 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.071 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.071 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.071 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:06.071 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:06.071 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.071 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.071 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.071 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:06.071 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.072 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.072 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:06.072 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:06.072 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.072 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.072 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.072 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:06.072 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:06.073 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.073 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.073 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:06.073 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.073 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.073 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:06.073 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:06.073 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.074 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.074 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.074 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:06.074 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:06.074 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.074 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.074 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.074 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:06.075 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.075 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.075 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:06.075 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:06.075 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.075 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.075 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.075 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.076 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:06.076 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:06.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.076 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.076 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.076 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:06.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.076 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.076 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:06.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.076 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.077 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.077 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:48:06.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.077 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.077 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:06.077 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:06.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.025+0000 7f2435597640 1 -- 172.21.15.64:0/2804375263 >> v1:172.21.15.100:6812/4210963484 conn(0x7f241c0405d0 legacy=0x7f241c042a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.025+0000 7f2435597640 10 monclient: shutdown 2024-09-17T15:48:06.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.025+0000 7f2435597640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:06.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.025+0000 7f2435597640 1 -- 172.21.15.64:0/2804375263 >> v1:172.21.15.64:6793/0 conn(0x7f2430135f70 legacy=0x7f24301d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.025+0000 7f242f7fe640 1 -- 172.21.15.64:0/2804375263 reap_dead start 2024-09-17T15:48:06.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.025+0000 7f242f7fe640 1 -- 172.21.15.64:0/2804375263 reap_dead start 2024-09-17T15:48:06.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.025+0000 7f2435597640 1 -- 172.21.15.64:0/2804375263 shutdown_connections 2024-09-17T15:48:06.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.025+0000 7f2435597640 1 -- 172.21.15.64:0/2804375263 >> 172.21.15.64:0/2804375263 conn(0x7f243006db30 msgr2=0x7f2430133dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:06.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.025+0000 7f2435597640 1 -- 172.21.15.64:0/2804375263 shutdown_connections 2024-09-17T15:48:06.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.025+0000 7f2435597640 1 -- 172.21.15.64:0/2804375263 wait complete. 2024-09-17T15:48:06.079 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:06.080 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-17T15:48:06.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-17T15:48:06.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 10 monclient: get_monmap_and_config 2024-09-17T15:48:06.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 10 monclient: build_initial_monmap 2024-09-17T15:48:06.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 10 monclient: monmap: 2024-09-17T15:48:06.182 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:06.181191+0000 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:06.181191+0000 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:06.183 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:06.184 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 1 Processor -- start 2024-09-17T15:48:06.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 1 -- start start 2024-09-17T15:48:06.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 10 monclient: init 2024-09-17T15:48:06.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:06.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 10 monclient: _add_conns ranks=[2,3,1,6,5,4,0,7,8] 2024-09-17T15:48:06.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 10 monclient(hunting): picked mon.noname-b con 0x7f4e040a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:06.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 10 monclient(hunting): picked mon.noname-g con 0x7f4e04135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:06.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 10 monclient(hunting): picked mon.noname-f con 0x7f4e04135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:06.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e0409d6f0 con 0x7f4e04135f70 2024-09-17T15:48:06.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e0409d9d0 con 0x7f4e040a5520 2024-09-17T15:48:06.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e0409dcb0 con 0x7f4e04135b50 2024-09-17T15:48:06.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:06.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e0ba33640 10 monclient(hunting): authenticate will time out at 883.856833s 2024-09-17T15:48:06.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e097cf640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f4e040a5520 0x7f4e040a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53014/0 (socket says 172.21.15.64:53014) 2024-09-17T15:48:06.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.180+0000 7f4e097cf640 1 -- 172.21.15.64:0/3536360034 learned_addr learned my addr 172.21.15.64:0/3536360034 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 -- 172.21.15.64:0/3536360034 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 423021053 0 0) 0x7f4e0409d9d0 con 0x7f4e040a5520 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient(hunting): my global_id is 4970 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 -- 172.21.15.64:0/3536360034 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df0003520 con 0x7f4e040a5520 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 -- 172.21.15.64:0/3536360034 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4209618084 0 0) 0x7f4e0409dcb0 con 0x7f4e04135b50 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient(hunting): my global_id is 114222 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 -- 172.21.15.64:0/3536360034 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e0409d9d0 con 0x7f4e04135b50 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 -- 172.21.15.64:0/3536360034 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3062266409 0 0) 0x7f4e0409d6f0 con 0x7f4e04135f70 2024-09-17T15:48:06.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient(hunting): my global_id is 5239 2024-09-17T15:48:06.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 -- 172.21.15.64:0/3536360034 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e0409dcb0 con 0x7f4e04135f70 2024-09-17T15:48:06.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 -- 172.21.15.64:0/3536360034 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 206395858 0 0) 0x7f4df0003520 con 0x7f4e040a5520 2024-09-17T15:48:06.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 -- 172.21.15.64:0/3536360034 >> v1:172.21.15.100:6790/0 conn(0x7f4e04135b50 legacy=0x7f4e0413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 -- 172.21.15.64:0/3536360034 >> v1:172.21.15.100:6789/0 conn(0x7f4e04135f70 legacy=0x7f4e041415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:48:06.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 monclient: found mon.noname-b 2024-09-17T15:48:06.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:06.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:48:06.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 -- 172.21.15.64:0/3536360034 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0409ccd0 con 0x7f4e040a5520 2024-09-17T15:48:06.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient: _finish_auth 0 2024-09-17T15:48:06.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:48:06.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:06.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 -- 172.21.15.64:0/3536360034 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4e00001830 con 0x7f4e040a5520 2024-09-17T15:48:06.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4e0ba33640 5 monclient: authenticate success, global_id 4970 2024-09-17T15:48:06.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:06.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient: dump: 2024-09-17T15:48:06.196 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.197 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.198 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4e0ba33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:06.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 -- 172.21.15.64:0/3536360034 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4e000019d0 con 0x7f4e040a5520 2024-09-17T15:48:06.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:06.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 1 -- 172.21.15.64:0/3536360034 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4e00004be0 con 0x7f4e040a5520 2024-09-17T15:48:06.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4e0ba33640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:06.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4e0ba33640 4 set_mon_vals no callback set 2024-09-17T15:48:06.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:06.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4df7fff640 10 monclient: dump: 2024-09-17T15:48:06.198 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.198 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.199 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.199 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.199 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.199 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.199 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.199 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.199 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.199 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.199 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.199 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.200 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.200 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.200 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.200 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4e0ba33640 10 monclient: shutdown 2024-09-17T15:48:06.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4e0ba33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:06.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.181+0000 7f4e0ba33640 1 -- 172.21.15.64:0/3536360034 >> v1:172.21.15.64:6790/0 conn(0x7f4e040a5520 legacy=0x7f4e040a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.182+0000 7f4e0ba33640 1 -- 172.21.15.64:0/3536360034 shutdown_connections 2024-09-17T15:48:06.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.182+0000 7f4e0ba33640 1 -- 172.21.15.64:0/3536360034 >> 172.21.15.64:0/3536360034 conn(0x7f4e0406db30 msgr2=0x7f4e040ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:06.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.182+0000 7f4e0ba33640 1 -- 172.21.15.64:0/3536360034 shutdown_connections 2024-09-17T15:48:06.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.182+0000 7f4e0ba33640 1 -- 172.21.15.64:0/3536360034 wait complete. 2024-09-17T15:48:06.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.182+0000 7f4e0ba33640 10 monclient: build_initial_monmap 2024-09-17T15:48:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.182+0000 7f4e0ba33640 10 monclient: monmap: 2024-09-17T15:48:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:06.201 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:06.202 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:06.202 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:06.202 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:06.202 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:06.202 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:06.202 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e0ba33640 1 Processor -- start 2024-09-17T15:48:06.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e0ba33640 1 -- start start 2024-09-17T15:48:06.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e0ba33640 10 monclient: init 2024-09-17T15:48:06.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e0ba33640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:06.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e0ba33640 10 monclient: _add_conns ranks=[3,5,4,0,8,7,1,2,6] 2024-09-17T15:48:06.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e0ba33640 10 monclient(hunting): picked mon.noname-g con 0x7f4e040a5520 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:06.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e0ba33640 10 monclient(hunting): picked mon.noname-h con 0x7f4e04135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:06.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e0ba33640 10 monclient(hunting): picked mon.noname-c con 0x7f4e04135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:06.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e0ba33640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e0413ae20 con 0x7f4e040a5520 2024-09-17T15:48:06.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e0ba33640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e0413a240 con 0x7f4e04135f70 2024-09-17T15:48:06.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e0ba33640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e04138ac0 con 0x7f4e04135b50 2024-09-17T15:48:06.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e0ba33640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:06.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e0ba33640 10 monclient(hunting): authenticate will time out at 883.859618s 2024-09-17T15:48:06.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e097cf640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f4e040a5520 0x7f4e04136fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:46632/0 (socket says 172.21.15.64:46632) 2024-09-17T15:48:06.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e097cf640 1 -- 172.21.15.64:0/2812632670 learned_addr learned my addr 172.21.15.64:0/2812632670 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:06.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4e09fd0640 1 --1- 172.21.15.64:0/2812632670 >> v1:172.21.15.64:6791/0 conn(0x7f4e04135f70 0x7f4e04137dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:58862/0 (socket says 172.21.15.64:58862) 2024-09-17T15:48:06.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 424023044 0 0) 0x7f4e0413ae20 con 0x7f4e040a5520 2024-09-17T15:48:06.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 10 monclient(hunting): my global_id is 114231 2024-09-17T15:48:06.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4de8003550 con 0x7f4e040a5520 2024-09-17T15:48:06.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2481546035 0 0) 0x7f4e04138ac0 con 0x7f4e04135b50 2024-09-17T15:48:06.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 10 monclient(hunting): my global_id is 114251 2024-09-17T15:48:06.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e0413ae20 con 0x7f4e04135b50 2024-09-17T15:48:06.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2481655870 0 0) 0x7f4e0413a240 con 0x7f4e04135f70 2024-09-17T15:48:06.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 10 monclient(hunting): my global_id is 5206 2024-09-17T15:48:06.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.183+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e04138ac0 con 0x7f4e04135f70 2024-09-17T15:48:06.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2041040952 0 0) 0x7f4de8003550 con 0x7f4e040a5520 2024-09-17T15:48:06.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4e0413a240 con 0x7f4e040a5520 2024-09-17T15:48:06.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4e000047e0 con 0x7f4e040a5520 2024-09-17T15:48:06.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:48:06.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient(hunting): dump: 2024-09-17T15:48:06.205 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.206 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.206 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.206 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.206 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.206 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.206 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.206 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.207 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.207 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.207 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.207 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.207 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.207 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.207 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.207 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:06.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 158135504 0 0) 0x7f4e0413ae20 con 0x7f4e04135b50 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4de8003550 con 0x7f4e04135b50 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4df8001530 con 0x7f4e04135b50 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient(hunting): dump: 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.208 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.209 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.209 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.209 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.209 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.209 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.209 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.209 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3998522116 0 0) 0x7f4e04138ac0 con 0x7f4e04135f70 2024-09-17T15:48:06.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4e0413ae20 con 0x7f4e04135f70 2024-09-17T15:48:06.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4dfc001cf0 con 0x7f4e04135f70 2024-09-17T15:48:06.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:06.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient(hunting): dump: 2024-09-17T15:48:06.210 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.210 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.210 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.210 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.210 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.210 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.210 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.210 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.210 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.210 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.211 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.211 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.211 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.211 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.211 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.211 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2570775486 0 0) 0x7f4e0413a240 con 0x7f4e040a5520 2024-09-17T15:48:06.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 >> v1:172.21.15.100:6791/0 conn(0x7f4e04135b50 legacy=0x7f4e041376c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 >> v1:172.21.15.64:6791/0 conn(0x7f4e04135f70 legacy=0x7f4e04137dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:48:06.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 monclient: found mon.g 2024-09-17T15:48:06.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:06.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:06.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0413a580 con 0x7f4e040a5520 2024-09-17T15:48:06.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:48:06.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:48:06.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:06.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4e0ba33640 5 monclient: authenticate success, global_id 114231 2024-09-17T15:48:06.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4e0ba33640 10 monclient: _renew_subs 2024-09-17T15:48:06.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4e0ba33640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:06.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4e0ba33640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0413a860 con 0x7f4e040a5520 2024-09-17T15:48:06.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.3 v1:172.21.15.100:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4e00004ea0 con 0x7f4e040a5520 2024-09-17T15:48:06.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:06.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.3 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4e00005390 con 0x7f4e040a5520 2024-09-17T15:48:06.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df5ffb640 10 monclient: dump: 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.213 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.214 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.214 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4df77fe640 4 set_mon_vals no callback set 2024-09-17T15:48:06.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4e0ba33640 10 monclient: _renew_subs 2024-09-17T15:48:06.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4e0ba33640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:06.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.184+0000 7f4e0ba33640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4e041d7160 con 0x7f4e040a5520 2024-09-17T15:48:06.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.3 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f4e00011c70 con 0x7f4e040a5520 2024-09-17T15:48:06.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4e0ba33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:06.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4e0ba33640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4e040008d0 con 0x7f4e04142f10 2024-09-17T15:48:06.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.3 v1:172.21.15.100:6790/0 8 ==== osd_map(112..112 src has 1..112) ==== 3898+0+0 (unknown 4054225456 0 0) 0x7f4e0004ec30 con 0x7f4e040a5520 2024-09-17T15:48:06.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2927738196 0 0) 0x7f4e040008d0 con 0x7f4e04142f10 2024-09-17T15:48:06.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4e09fd0640 10 monclient: get_auth_request con 0x7f4de8040da0 auth_method 0 2024-09-17T15:48:06.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 10 monclient(hunting): my global_id is 5156 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e0413a240 con 0x7f4e04142f10 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2256957089 0 0) 0x7f4e0413a240 con 0x7f4e04142f10 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4e040008d0 con 0x7f4e04142f10 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4df8003970 con 0x7f4e04142f10 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.185+0000 7f4df5ffb640 10 monclient: dump: 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.215 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.216 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.216 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.216 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.216 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.216 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.216 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.216 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.216 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.216 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.216 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.216 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.186+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1123207256 0 0) 0x7f4e040008d0 con 0x7f4e04142f10 2024-09-17T15:48:06.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.186+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.64:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4e04134f50 con 0x7f4e04142f10 2024-09-17T15:48:06.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.186+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4df8003d10 con 0x7f4e04142f10 2024-09-17T15:48:06.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.186+0000 7f4df5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:06.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.186+0000 7f4df5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:06.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.186+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 >> v1:172.21.15.64:6793/0 conn(0x7f4e04142f10 legacy=0x7f4e040a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.186+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 >> v1:172.21.15.64:6793/0 conn(0x7f4e04142f10 legacy=0x7f4e040a0480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:06.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.192+0000 7f4e0ba33640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:06.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.192+0000 7f4e0ba33640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4e0402e830 con 0x7f4e0402e450 2024-09-17T15:48:06.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.192+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 914704414 0 0) 0x7f4e0402e830 con 0x7f4e0402e450 2024-09-17T15:48:06.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.192+0000 7f4df5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.192+0000 7f4df5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.192+0000 7f4df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.192+0000 7f4df5ffb640 10 monclient(hunting): my global_id is 5165 2024-09-17T15:48:06.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.192+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e040008d0 con 0x7f4e0402e450 2024-09-17T15:48:06.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.193+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4097846953 0 0) 0x7f4e040008d0 con 0x7f4e0402e450 2024-09-17T15:48:06.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.193+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4e0402e830 con 0x7f4e0402e450 2024-09-17T15:48:06.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.193+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4df80044b0 con 0x7f4e0402e450 2024-09-17T15:48:06.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.193+0000 7f4df5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.193+0000 7f4df5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:06.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.193+0000 7f4df5ffb640 10 monclient: dump: 2024-09-17T15:48:06.220 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.221 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.221 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.221 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.221 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.221 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.221 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.222 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.222 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.222 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.222 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.222 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.222 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.223 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.223 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.223 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.193+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1383985817 0 0) 0x7f4e0402e830 con 0x7f4e0402e450 2024-09-17T15:48:06.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.193+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 --> v1:172.21.15.64:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4e040a12a0 con 0x7f4e0402e450 2024-09-17T15:48:06.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.193+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1377913796) 0x7f4df8005960 con 0x7f4e0402e450 2024-09-17T15:48:06.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.193+0000 7f4df5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:06.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.193+0000 7f4df5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:06.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.193+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 >> v1:172.21.15.64:6793/0 conn(0x7f4e0402e450 legacy=0x7f4e040a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.193+0000 7f4df5ffb640 1 -- 172.21.15.64:0/2812632670 >> v1:172.21.15.64:6793/0 conn(0x7f4e0402e450 legacy=0x7f4e040a0b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:06.224 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:06.224 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:06.224 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:06.224 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:06.224 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:06.224 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 142841, 2024-09-17T15:48:06.224 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:06.225 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:06.226 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.227 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:06.228 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:06.229 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:06.229 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:06.229 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.229 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.229 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.229 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.229 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.229 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.229 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.230 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.230 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.230 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.230 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.230 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.230 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:06.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.231 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:06.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.231 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:06.231 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:06.232 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.232 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.232 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:06.232 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.233 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.233 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:06.233 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:06.233 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.233 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.234 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.234 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:06.234 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:06.234 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:06.235 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.236 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.236 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:06.236 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:06.236 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.237 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.237 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.237 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:06.237 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:06.238 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.238 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.238 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.238 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:06.238 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.239 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.239 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:06.239 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:06.239 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.239 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.239 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.240 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:06.240 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:06.240 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.240 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:06.241 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.241 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:06.242 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:06.242 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.242 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.242 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.242 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:06.243 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:06.243 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.243 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.243 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.243 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:06.243 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.244 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.244 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:06.244 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:06.244 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.244 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.244 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.245 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:06.245 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:06.245 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.245 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.246 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.246 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:06.246 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.246 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.246 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:06.246 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:06.247 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.247 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.247 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.247 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:06.248 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:06.248 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.248 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.248 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.248 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:06.249 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.249 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.249 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:06.249 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:06.250 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.250 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.250 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.250 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:06.251 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:06.251 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.251 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.251 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.251 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:06.251 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.252 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.252 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:06.252 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:06.252 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.252 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.252 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.253 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:06.253 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:06.253 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.253 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.254 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:06.254 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.254 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.254 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:06.254 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:06.255 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.255 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.255 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.255 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.255 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:06.256 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:06.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.256 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.256 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.256 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:06.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.256 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.256 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:48:06.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.256 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.256 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.257 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:48:06.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.257 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.257 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:06.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.257 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.257 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.257 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:06.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.257 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:06.257 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:06.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.194+0000 7f4e0ba33640 1 -- 172.21.15.64:0/2812632670 >> v1:172.21.15.100:6812/4210963484 conn(0x7f4de8040da0 legacy=0x7f4de8043260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.194+0000 7f4e0ba33640 10 monclient: shutdown 2024-09-17T15:48:06.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.194+0000 7f4e0ba33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:06.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.194+0000 7f4e0ba33640 1 -- 172.21.15.64:0/2812632670 >> v1:172.21.15.100:6790/0 conn(0x7f4e040a5520 legacy=0x7f4e04136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.194+0000 7f4e09fd0640 1 -- 172.21.15.64:0/2812632670 reap_dead start 2024-09-17T15:48:06.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.194+0000 7f4e09fd0640 1 -- 172.21.15.64:0/2812632670 reap_dead start 2024-09-17T15:48:06.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.194+0000 7f4e0ba33640 1 -- 172.21.15.64:0/2812632670 shutdown_connections 2024-09-17T15:48:06.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.194+0000 7f4e0ba33640 1 -- 172.21.15.64:0/2812632670 >> 172.21.15.64:0/2812632670 conn(0x7f4e0406db30 msgr2=0x7f4e04133e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:06.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.194+0000 7f4e0ba33640 1 -- 172.21.15.64:0/2812632670 shutdown_connections 2024-09-17T15:48:06.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.194+0000 7f4e0ba33640 1 -- 172.21.15.64:0/2812632670 wait complete. 2024-09-17T15:48:06.260 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:06.260 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-17T15:48:06.260 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-17T15:48:06.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.360+0000 7fa7ea124640 10 monclient: get_monmap_and_config 2024-09-17T15:48:06.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.360+0000 7fa7ea124640 10 monclient: build_initial_monmap 2024-09-17T15:48:06.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.360+0000 7fa7ea124640 10 monclient: monmap: 2024-09-17T15:48:06.362 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:06.362 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.363 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:06.361658+0000 2024-09-17T15:48:06.363 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:06.361658+0000 2024-09-17T15:48:06.363 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:06.363 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.363 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:06.363 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:06.363 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:06.363 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:06.363 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:06.363 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:06.363 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:06.363 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:06.363 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:06.364 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.360+0000 7fa7ea124640 1 Processor -- start 2024-09-17T15:48:06.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.360+0000 7fa7ea124640 1 -- start start 2024-09-17T15:48:06.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.360+0000 7fa7ea124640 10 monclient: init 2024-09-17T15:48:06.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7ea124640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:06.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7ea124640 10 monclient: _add_conns ranks=[7,5,3,4,2,6,8,0,1] 2024-09-17T15:48:06.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7ea124640 10 monclient(hunting): picked mon.noname-i con 0x7fa7e40a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:06.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7ea124640 10 monclient(hunting): picked mon.noname-h con 0x7fa7e4135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:06.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7ea124640 10 monclient(hunting): picked mon.noname-g con 0x7fa7e4135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:06.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7ea124640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e409d6f0 con 0x7fa7e4135f70 2024-09-17T15:48:06.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7ea124640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e409d9d0 con 0x7fa7e4135b50 2024-09-17T15:48:06.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7ea124640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e409dcb0 con 0x7fa7e40a5520 2024-09-17T15:48:06.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7ea124640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:06.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7ea124640 10 monclient(hunting): authenticate will time out at 884.037357s 2024-09-17T15:48:06.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e2ffd640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7fa7e4135b50 0x7fa7e413deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:47430/0 (socket says 172.21.15.64:47430) 2024-09-17T15:48:06.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e2ffd640 1 -- 172.21.15.64:0/1036972171 learned_addr learned my addr 172.21.15.64:0/1036972171 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:06.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 1 -- 172.21.15.64:0/1036972171 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3402177044 0 0) 0x7fa7e409d6f0 con 0x7fa7e4135f70 2024-09-17T15:48:06.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient(hunting): my global_id is 114240 2024-09-17T15:48:06.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 1 -- 172.21.15.64:0/1036972171 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7cc003520 con 0x7fa7e4135f70 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 1 -- 172.21.15.64:0/1036972171 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4077925567 0 0) 0x7fa7e409d9d0 con 0x7fa7e4135b50 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient(hunting): my global_id is 114260 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 1 -- 172.21.15.64:0/1036972171 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e409d6f0 con 0x7fa7e4135b50 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 1 -- 172.21.15.64:0/1036972171 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4267293712 0 0) 0x7fa7e409dcb0 con 0x7fa7e40a5520 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient(hunting): my global_id is 5074 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 1 -- 172.21.15.64:0/1036972171 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e409d9d0 con 0x7fa7e40a5520 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 1 -- 172.21.15.64:0/1036972171 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 667110375 0 0) 0x7fa7cc003520 con 0x7fa7e4135f70 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 1 -- 172.21.15.64:0/1036972171 >> v1:172.21.15.100:6792/0 conn(0x7fa7e40a5520 legacy=0x7fa7e40a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 1 -- 172.21.15.64:0/1036972171 >> v1:172.21.15.100:6791/0 conn(0x7fa7e4135b50 legacy=0x7fa7e413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:06.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 1 monclient: found mon.noname-g 2024-09-17T15:48:06.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:06.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.100:6790/0 2024-09-17T15:48:06.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 1 -- 172.21.15.64:0/1036972171 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e409ccd0 con 0x7fa7e4135f70 2024-09-17T15:48:06.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient: _finish_auth 0 2024-09-17T15:48:06.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:48:06.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.361+0000 7fa7e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:06.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7ea124640 5 monclient: authenticate success, global_id 114240 2024-09-17T15:48:06.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7ea124640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:06.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7e27fc640 1 -- 172.21.15.64:0/1036972171 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa7dc001bb0 con 0x7fa7e4135f70 2024-09-17T15:48:06.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7e27fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:48:06.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7e27fc640 10 monclient: dump: 2024-09-17T15:48:06.376 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.376 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.376 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.376 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.376 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.376 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.376 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.376 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.376 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.376 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.376 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.376 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7ea124640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7e27fc640 1 -- 172.21.15.64:0/1036972171 <== mon.3 v1:172.21.15.100:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7dc001e10 con 0x7fa7e4135f70 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7e27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7ea124640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7ea124640 4 set_mon_vals no callback set 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7e27fc640 1 -- 172.21.15.64:0/1036972171 <== mon.3 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa7dc003cd0 con 0x7fa7e4135f70 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7e27fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7e27fc640 10 monclient: dump: 2024-09-17T15:48:06.377 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7ea124640 10 monclient: shutdown 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7ea124640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7ea124640 1 -- 172.21.15.64:0/1036972171 >> v1:172.21.15.100:6790/0 conn(0x7fa7e4135f70 legacy=0x7fa7e41415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7ea124640 1 -- 172.21.15.64:0/1036972171 shutdown_connections 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7ea124640 1 -- 172.21.15.64:0/1036972171 >> 172.21.15.64:0/1036972171 conn(0x7fa7e406db30 msgr2=0x7fa7e40ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7ea124640 1 -- 172.21.15.64:0/1036972171 shutdown_connections 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.362+0000 7fa7ea124640 1 -- 172.21.15.64:0/1036972171 wait complete. 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.363+0000 7fa7ea124640 10 monclient: build_initial_monmap 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.363+0000 7fa7ea124640 10 monclient: monmap: 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:06.379 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.363+0000 7fa7ea124640 1 Processor -- start 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.363+0000 7fa7ea124640 1 -- start start 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.363+0000 7fa7ea124640 10 monclient: init 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7ea124640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:06.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7ea124640 10 monclient: _add_conns ranks=[2,4,3,1,8,0,5,7,6] 2024-09-17T15:48:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7ea124640 10 monclient(hunting): picked mon.noname-b con 0x7fa7e40a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7ea124640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e4135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7ea124640 10 monclient(hunting): picked mon.noname-g con 0x7fa7e4135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7ea124640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e40a63e0 con 0x7fa7e40a5520 2024-09-17T15:48:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7ea124640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e413a320 con 0x7fa7e4135f70 2024-09-17T15:48:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7ea124640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e4138c10 con 0x7fa7e4135b50 2024-09-17T15:48:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7ea124640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7ea124640 10 monclient(hunting): authenticate will time out at 884.040285s 2024-09-17T15:48:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7e2ffd640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fa7e4135b50 0x7fa7e41378b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:58874/0 (socket says 172.21.15.64:58874) 2024-09-17T15:48:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7e2ffd640 1 -- 172.21.15.64:0/2333242791 learned_addr learned my addr 172.21.15.64:0/2333242791 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7e37fe640 1 --1- 172.21.15.64:0/2333242791 >> v1:172.21.15.64:6790/0 conn(0x7fa7e40a5520 0x7fa7e41371a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53018/0 (socket says 172.21.15.64:53018) 2024-09-17T15:48:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 243133285 0 0) 0x7fa7e4138c10 con 0x7fa7e4135b50 2024-09-17T15:48:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): my global_id is 5215 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7d0003550 con 0x7fa7e4135b50 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 91409789 0 0) 0x7fa7e40a63e0 con 0x7fa7e40a5520 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): my global_id is 4979 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e4138c10 con 0x7fa7e40a5520 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3827310423 0 0) 0x7fa7e413a320 con 0x7fa7e4135f70 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): my global_id is 114249 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e40a63e0 con 0x7fa7e4135f70 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2653428196 0 0) 0x7fa7e4138c10 con 0x7fa7e40a5520 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7e413a320 con 0x7fa7e40a5520 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3603939275 0 0) 0x7fa7d0003550 con 0x7fa7e4135b50 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7e4138c10 con 0x7fa7e4135b50 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa7d4001a00 con 0x7fa7e40a5520 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): dump: 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.383 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa7d8001a90 con 0x7fa7e4135b50 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.364+0000 7fa7cbfff640 10 monclient(hunting): dump: 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.385 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3584129582 0 0) 0x7fa7e40a63e0 con 0x7fa7e4135f70 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7d0003550 con 0x7fa7e4135f70 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa7dc001af0 con 0x7fa7e4135f70 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 10 monclient(hunting): dump: 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.386 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.387 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.387 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.387 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.387 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.387 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.387 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.387 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.387 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.387 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1257962006 0 0) 0x7fa7e413a320 con 0x7fa7e40a5520 2024-09-17T15:48:06.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 >> v1:172.21.15.64:6791/0 conn(0x7fa7e4135b50 legacy=0x7fa7e41378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 >> v1:172.21.15.100:6790/0 conn(0x7fa7e4135f70 legacy=0x7fa7e4137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 10 monclient: _finish_hunting 0 2024-09-17T15:48:06.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 1 monclient: found mon.b 2024-09-17T15:48:06.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:06.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:06.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e413a6d0 con 0x7fa7e40a5520 2024-09-17T15:48:06.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 10 monclient: _finish_auth 0 2024-09-17T15:48:06.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:48:06.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:06.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7ea124640 5 monclient: authenticate success, global_id 4979 2024-09-17T15:48:06.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7ea124640 10 monclient: _renew_subs 2024-09-17T15:48:06.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7ea124640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:06.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7ea124640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e413a9b0 con 0x7fa7e40a5520 2024-09-17T15:48:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7ea124640 10 monclient: _renew_subs 2024-09-17T15:48:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7ea124640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7ea124640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa7e41d7160 con 0x7fa7e40a5520 2024-09-17T15:48:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7d4001ce0 con 0x7fa7e40a5520 2024-09-17T15:48:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa7d4003c00 con 0x7fa7e40a5520 2024-09-17T15:48:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7e1ffb640 4 set_mon_vals no callback set 2024-09-17T15:48:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.365+0000 7fa7cbfff640 10 monclient: dump: 2024-09-17T15:48:06.389 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.390 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7ea124640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7ea124640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa7c40065c0 con 0x7fa7c4002860 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fa7d4003da0 con 0x7fa7e40a5520 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(112..112 src has 1..112) ==== 3898+0+0 (unknown 4054225456 0 0) 0x7fa7d404ebd0 con 0x7fa7e40a5520 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750872493 0 0) 0x7fa7c40065c0 con 0x7fa7c4002860 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7e3fff640 10 monclient: get_auth_request con 0x7fa7d0040ca0 auth_method 0 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 10 monclient(hunting): my global_id is 5248 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e413a320 con 0x7fa7c4002860 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3707259479 0 0) 0x7fa7e413a320 con 0x7fa7c4002860 2024-09-17T15:48:06.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7c40065c0 con 0x7fa7c4002860 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa7d8004fc0 con 0x7fa7c4002860 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.366+0000 7fa7cbfff640 10 monclient: dump: 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.392 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.393 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.367+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 116829173 0 0) 0x7fa7c40065c0 con 0x7fa7c4002860 2024-09-17T15:48:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.367+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.100:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7c400a160 con 0x7fa7c4002860 2024-09-17T15:48:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.367+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa7d8006540 con 0x7fa7c4002860 2024-09-17T15:48:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.367+0000 7fa7cbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.367+0000 7fa7cbfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.367+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 >> v1:172.21.15.100:6789/0 conn(0x7fa7c4002860 legacy=0x7fa7c4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.367+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 >> v1:172.21.15.100:6789/0 conn(0x7fa7c4002860 legacy=0x7fa7c4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:06.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.373+0000 7fa7ea124640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:06.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.373+0000 7fa7ea124640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa7c4009e00 con 0x7fa7c40090e0 2024-09-17T15:48:06.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.373+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3067313943 0 0) 0x7fa7c4009e00 con 0x7fa7c40090e0 2024-09-17T15:48:06.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.373+0000 7fa7cbfff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.373+0000 7fa7cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.373+0000 7fa7cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.373+0000 7fa7cbfff640 10 monclient(hunting): my global_id is 5257 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.373+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7c40065c0 con 0x7fa7c40090e0 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3043714385 0 0) 0x7fa7c40065c0 con 0x7fa7c40090e0 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7c4009e00 con 0x7fa7c40090e0 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa7d8005630 con 0x7fa7c40090e0 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7cbfff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7cbfff640 10 monclient: dump: 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.396 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.397 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.397 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.397 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.397 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.397 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.397 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.397 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.397 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1157464414 0 0) 0x7fa7c4009e00 con 0x7fa7c40090e0 2024-09-17T15:48:06.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 --> v1:172.21.15.100:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7c400c3a0 con 0x7fa7c40090e0 2024-09-17T15:48:06.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 2765918427) 0x7fa7d8005980 con 0x7fa7c40090e0 2024-09-17T15:48:06.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7cbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:06.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7cbfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:06.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 >> v1:172.21.15.100:6789/0 conn(0x7fa7c40090e0 legacy=0x7fa7c4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7cbfff640 1 -- 172.21.15.64:0/2333242791 >> v1:172.21.15.100:6789/0 conn(0x7fa7c40090e0 legacy=0x7fa7c4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:06.398 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 143025, 2024-09-17T15:48:06.398 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:06.399 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.400 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.401 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:06.402 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.403 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:06.404 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:06.405 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:06.407 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:06.408 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.409 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:06.410 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.412 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:06.413 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.414 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:06.415 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.416 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:48:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:06.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:48:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.418 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:06.418 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:06.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7ea124640 1 -- 172.21.15.64:0/2333242791 >> v1:172.21.15.100:6812/4210963484 conn(0x7fa7d0040ca0 legacy=0x7fa7d0043160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7ea124640 10 monclient: shutdown 2024-09-17T15:48:06.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7ea124640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:06.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7ea124640 1 -- 172.21.15.64:0/2333242791 >> v1:172.21.15.64:6790/0 conn(0x7fa7e40a5520 legacy=0x7fa7e41371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7e3fff640 1 -- 172.21.15.64:0/2333242791 reap_dead start 2024-09-17T15:48:06.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7e3fff640 1 -- 172.21.15.64:0/2333242791 reap_dead start 2024-09-17T15:48:06.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7ea124640 1 -- 172.21.15.64:0/2333242791 shutdown_connections 2024-09-17T15:48:06.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.374+0000 7fa7ea124640 1 -- 172.21.15.64:0/2333242791 >> 172.21.15.64:0/2333242791 conn(0x7fa7e406db30 msgr2=0x7fa7e40a4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:06.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.375+0000 7fa7ea124640 1 -- 172.21.15.64:0/2333242791 shutdown_connections 2024-09-17T15:48:06.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.375+0000 7fa7ea124640 1 -- 172.21.15.64:0/2333242791 wait complete. 2024-09-17T15:48:06.421 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:06.421 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-17T15:48:06.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-17T15:48:06.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 10 monclient: get_monmap_and_config 2024-09-17T15:48:06.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 10 monclient: build_initial_monmap 2024-09-17T15:48:06.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 10 monclient: monmap: 2024-09-17T15:48:06.523 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:06.523 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.523 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:06.522306+0000 2024-09-17T15:48:06.524 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:06.522306+0000 2024-09-17T15:48:06.524 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:06.524 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.524 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:06.524 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:06.524 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:06.524 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:06.524 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:06.524 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:06.524 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:06.524 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:06.524 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:06.524 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 1 Processor -- start 2024-09-17T15:48:06.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 1 -- start start 2024-09-17T15:48:06.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 10 monclient: init 2024-09-17T15:48:06.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:06.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 10 monclient: _add_conns ranks=[6,0,2,8,7,5,1,4,3] 2024-09-17T15:48:06.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 10 monclient(hunting): picked mon.noname-d con 0x7f4aa40a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:06.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa4135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:06.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 10 monclient(hunting): picked mon.noname-b con 0x7f4aa4135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:06.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aa409d6f0 con 0x7f4aa4135b50 2024-09-17T15:48:06.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aa409d9d0 con 0x7f4aa4135f70 2024-09-17T15:48:06.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aa409dcb0 con 0x7f4aa40a5520 2024-09-17T15:48:06.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:06.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.521+0000 7f4aabb15640 10 monclient(hunting): authenticate will time out at 884.197987s 2024-09-17T15:48:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa98b1640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f4aa40a5520 0x7f4aa40a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:50346/0 (socket says 172.21.15.64:50346) 2024-09-17T15:48:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa98b1640 1 -- 172.21.15.64:0/3197940722 learned_addr learned my addr 172.21.15.64:0/3197940722 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 -- 172.21.15.64:0/3197940722 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3934707023 0 0) 0x7f4aa409dcb0 con 0x7f4aa40a5520 2024-09-17T15:48:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient(hunting): my global_id is 114252 2024-09-17T15:48:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 -- 172.21.15.64:0/3197940722 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a90003520 con 0x7f4aa40a5520 2024-09-17T15:48:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 -- 172.21.15.64:0/3197940722 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2265725055 0 0) 0x7f4aa409d9d0 con 0x7f4aa4135f70 2024-09-17T15:48:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient(hunting): my global_id is 4988 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 -- 172.21.15.64:0/3197940722 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4aa409dcb0 con 0x7f4aa4135f70 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 -- 172.21.15.64:0/3197940722 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1184176959 0 0) 0x7f4aa409d6f0 con 0x7f4aa4135b50 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient(hunting): my global_id is 5121 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 -- 172.21.15.64:0/3197940722 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4aa409d9d0 con 0x7f4aa4135b50 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 -- 172.21.15.64:0/3197940722 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 657725596 0 0) 0x7f4aa409dcb0 con 0x7f4aa4135f70 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 -- 172.21.15.64:0/3197940722 >> v1:172.21.15.64:6792/0 conn(0x7f4aa40a5520 legacy=0x7f4aa40a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 -- 172.21.15.64:0/3197940722 >> v1:172.21.15.64:6789/0 conn(0x7f4aa4135b50 legacy=0x7f4aa413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient: _finish_hunting 0 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 monclient: found mon.noname-b 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:06.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:48:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 -- 172.21.15.64:0/3197940722 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa409ccd0 con 0x7f4aa4135f70 2024-09-17T15:48:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient: _finish_auth 0 2024-09-17T15:48:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient: _check_auth_tickets 2024-09-17T15:48:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 -- 172.21.15.64:0/3197940722 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a9c001b90 con 0x7f4aa4135f70 2024-09-17T15:48:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aabb15640 5 monclient: authenticate success, global_id 4988 2024-09-17T15:48:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aabb15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient: dump: 2024-09-17T15:48:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.536 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aabb15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:06.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 -- 172.21.15.64:0/3197940722 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4a9c001d30 con 0x7f4aa4135f70 2024-09-17T15:48:06.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:06.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 1 -- 172.21.15.64:0/3197940722 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a9c003cd0 con 0x7f4aa4135f70 2024-09-17T15:48:06.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aabb15640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:06.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aabb15640 4 set_mon_vals no callback set 2024-09-17T15:48:06.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:06.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.522+0000 7f4aa88af640 10 monclient: dump: 2024-09-17T15:48:06.538 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.538 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.538 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.538 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.538 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.539 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.539 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.539 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.539 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.539 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.539 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.539 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.539 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.539 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.539 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.539 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.523+0000 7f4aabb15640 10 monclient: shutdown 2024-09-17T15:48:06.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.523+0000 7f4aabb15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:06.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.523+0000 7f4aabb15640 1 -- 172.21.15.64:0/3197940722 >> v1:172.21.15.64:6790/0 conn(0x7f4aa4135f70 legacy=0x7f4aa41415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.523+0000 7f4aabb15640 1 -- 172.21.15.64:0/3197940722 shutdown_connections 2024-09-17T15:48:06.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.523+0000 7f4aabb15640 1 -- 172.21.15.64:0/3197940722 >> 172.21.15.64:0/3197940722 conn(0x7f4aa406db30 msgr2=0x7f4aa40ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:06.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.523+0000 7f4aabb15640 1 -- 172.21.15.64:0/3197940722 shutdown_connections 2024-09-17T15:48:06.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.523+0000 7f4aabb15640 1 -- 172.21.15.64:0/3197940722 wait complete. 2024-09-17T15:48:06.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.523+0000 7f4aabb15640 10 monclient: build_initial_monmap 2024-09-17T15:48:06.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.523+0000 7f4aabb15640 10 monclient: monmap: 2024-09-17T15:48:06.540 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:06.540 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:06.540 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:06.540 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:06.540 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:06.540 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:06.541 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:06.541 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:06.541 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:06.541 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:06.541 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:06.541 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:06.541 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:06.541 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:06.541 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:06.541 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aabb15640 1 Processor -- start 2024-09-17T15:48:06.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aabb15640 1 -- start start 2024-09-17T15:48:06.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aabb15640 10 monclient: init 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aabb15640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aabb15640 10 monclient: _add_conns ranks=[0,4,7,3,2,8,5,6,1] 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aabb15640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa40a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aabb15640 10 monclient(hunting): picked mon.noname-c con 0x7f4aa4135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aabb15640 10 monclient(hunting): picked mon.noname-i con 0x7f4aa4135f70 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aabb15640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aa40a8f70 con 0x7f4aa40a5520 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aabb15640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aa4135310 con 0x7f4aa4135b50 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aabb15640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aa4134e00 con 0x7f4aa4135f70 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aabb15640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aabb15640 10 monclient(hunting): authenticate will time out at 884.200819s 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aa98b1640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4aa40a5520 0x7f4aa41cd4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:39844/0 (socket says 172.21.15.64:39844) 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.524+0000 7f4aa98b1640 1 -- 172.21.15.64:0/158474099 learned_addr learned my addr 172.21.15.64:0/158474099 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 747991844 0 0) 0x7f4aa0001b30 con 0x7f4aa40a5520 2024-09-17T15:48:06.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): my global_id is 5130 2024-09-17T15:48:06.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a88003550 con 0x7f4aa40a5520 2024-09-17T15:48:06.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1975448842 0 0) 0x7f4aa4135310 con 0x7f4aa4135b50 2024-09-17T15:48:06.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): my global_id is 5224 2024-09-17T15:48:06.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4aa0001b30 con 0x7f4aa4135b50 2024-09-17T15:48:06.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 222219709 0 0) 0x7f4aa4134e00 con 0x7f4aa4135f70 2024-09-17T15:48:06.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): my global_id is 5083 2024-09-17T15:48:06.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4aa4135310 con 0x7f4aa4135f70 2024-09-17T15:48:06.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 830686231 0 0) 0x7f4a88003550 con 0x7f4aa40a5520 2024-09-17T15:48:06.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4aa4134e00 con 0x7f4aa40a5520 2024-09-17T15:48:06.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4aa0001890 con 0x7f4aa40a5520 2024-09-17T15:48:06.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:06.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): dump: 2024-09-17T15:48:06.544 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.544 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.544 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.544 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.545 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.545 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.545 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.545 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.545 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.545 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.545 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.545 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.545 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.545 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.545 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.545 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:06.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3126721569 0 0) 0x7f4aa0001b30 con 0x7f4aa4135b50 2024-09-17T15:48:06.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a88003550 con 0x7f4aa4135b50 2024-09-17T15:48:06.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a98001210 con 0x7f4aa4135b50 2024-09-17T15:48:06.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:06.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): dump: 2024-09-17T15:48:06.546 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.546 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.546 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.546 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.546 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.546 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.546 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.547 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.547 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.547 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.547 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.547 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.547 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.547 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.547 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.547 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1211363584 0 0) 0x7f4aa4135310 con 0x7f4aa4135f70 2024-09-17T15:48:06.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4aa0001b30 con 0x7f4aa4135f70 2024-09-17T15:48:06.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a9c001b50 con 0x7f4aa4135f70 2024-09-17T15:48:06.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:48:06.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient(hunting): dump: 2024-09-17T15:48:06.548 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.548 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.548 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.548 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.548 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.548 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.548 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.548 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.548 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.548 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2671174998 0 0) 0x7f4aa4134e00 con 0x7f4aa40a5520 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 >> v1:172.21.15.100:6792/0 conn(0x7f4aa4135f70 legacy=0x7f4aa41d5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 >> v1:172.21.15.64:6791/0 conn(0x7f4aa4135b50 legacy=0x7f4aa40a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 monclient: found mon.a 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:48:06.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa4132fa0 con 0x7f4aa40a5520 2024-09-17T15:48:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient: _finish_auth 0 2024-09-17T15:48:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient: _check_auth_tickets 2024-09-17T15:48:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4aabb15640 5 monclient: authenticate success, global_id 5130 2024-09-17T15:48:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4aabb15640 10 monclient: _renew_subs 2024-09-17T15:48:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4aabb15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:48:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4aabb15640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa40a0100 con 0x7f4aa40a5520 2024-09-17T15:48:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4aabb15640 10 monclient: _renew_subs 2024-09-17T15:48:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4aabb15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:48:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4aabb15640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4aa40a5050 con 0x7f4aa40a5520 2024-09-17T15:48:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4aa0004a90 con 0x7f4aa40a5520 2024-09-17T15:48:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4aa0005040 con 0x7f4aa40a5520 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a97fff640 4 set_mon_vals no callback set 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.525+0000 7f4a967fc640 10 monclient: dump: 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.551 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.552 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.552 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.552 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.552 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.552 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.552 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.526+0000 7f4aabb15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:06.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.526+0000 7f4aabb15640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4a8c0065c0 con 0x7f4a8c002860 2024-09-17T15:48:06.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.526+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f4aa00051e0 con 0x7f4aa40a5520 2024-09-17T15:48:06.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(112..112 src has 1..112) ==== 3898+0+0 (unknown 4054225456 0 0) 0x7f4aa004eda0 con 0x7f4aa40a5520 2024-09-17T15:48:06.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2873558255 0 0) 0x7f4a8c0065c0 con 0x7f4a8c002860 2024-09-17T15:48:06.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 10 monclient(hunting): my global_id is 114258 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4aa4134e00 con 0x7f4a8c002860 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4aaa0b2640 10 monclient: get_auth_request con 0x7f4a88040c80 auth_method 0 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1715472803 0 0) 0x7f4aa4134e00 con 0x7f4a8c002860 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a8c0065c0 con 0x7f4a8c002860 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a98003930 con 0x7f4a8c002860 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 10 monclient: dump: 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.553 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.554 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.554 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3890945796 0 0) 0x7f4a8c0065c0 con 0x7f4a8c002860 2024-09-17T15:48:06.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.527+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.100:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a8c00a160 con 0x7f4a8c002860 2024-09-17T15:48:06.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.528+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4a98003ad0 con 0x7f4a8c002860 2024-09-17T15:48:06.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.528+0000 7f4a967fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:06.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.528+0000 7f4a967fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:06.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.528+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 >> v1:172.21.15.100:6790/0 conn(0x7f4a8c002860 legacy=0x7f4a8c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.528+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 >> v1:172.21.15.100:6790/0 conn(0x7f4a8c002860 legacy=0x7f4a8c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:06.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4aabb15640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:06.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4aabb15640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4a8c009e70 con 0x7f4a8c009130 2024-09-17T15:48:06.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2845399336 0 0) 0x7f4a8c009e70 con 0x7f4a8c009130 2024-09-17T15:48:06.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4a967fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4a967fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4a967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4a967fc640 10 monclient(hunting): my global_id is 114267 2024-09-17T15:48:06.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a8c0065c0 con 0x7f4a8c009130 2024-09-17T15:48:06.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2005743228 0 0) 0x7f4a8c0065c0 con 0x7f4a8c009130 2024-09-17T15:48:06.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a8c009e70 con 0x7f4a8c009130 2024-09-17T15:48:06.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a98004340 con 0x7f4a8c009130 2024-09-17T15:48:06.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4a967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4a967fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:06.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.534+0000 7f4a967fc640 10 monclient: dump: 2024-09-17T15:48:06.557 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.558 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1303960858 0 0) 0x7f4a8c009e70 con 0x7f4a8c009130 2024-09-17T15:48:06.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 --> v1:172.21.15.100:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4a8c00c3f0 con 0x7f4a8c009130 2024-09-17T15:48:06.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 562866251) 0x7f4a980065c0 con 0x7f4a8c009130 2024-09-17T15:48:06.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4a967fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:06.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4a967fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:06.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 >> v1:172.21.15.100:6790/0 conn(0x7f4a8c009130 legacy=0x7f4a8c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4a967fc640 1 -- 172.21.15.64:0/158474099 >> v1:172.21.15.100:6790/0 conn(0x7f4a8c009130 legacy=0x7f4a8c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:06.559 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:06.559 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 9758, 2024-09-17T15:48:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:06.560 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.561 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.562 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:06.563 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.564 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:06.565 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.566 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:06.567 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.570 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.572 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.574 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:06.576 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.578 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:06.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.581 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:06.581 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4aabb15640 1 -- 172.21.15.64:0/158474099 >> v1:172.21.15.100:6812/4210963484 conn(0x7f4a88040c80 legacy=0x7f4a88043140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4aabb15640 10 monclient: shutdown 2024-09-17T15:48:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4aabb15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4aabb15640 1 -- 172.21.15.64:0/158474099 >> v1:172.21.15.64:6789/0 conn(0x7f4aa40a5520 legacy=0x7f4aa41cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4aaa0b2640 1 -- 172.21.15.64:0/158474099 reap_dead start 2024-09-17T15:48:06.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4aaa0b2640 1 -- 172.21.15.64:0/158474099 reap_dead start 2024-09-17T15:48:06.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4aabb15640 1 -- 172.21.15.64:0/158474099 shutdown_connections 2024-09-17T15:48:06.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4aabb15640 1 -- 172.21.15.64:0/158474099 >> 172.21.15.64:0/158474099 conn(0x7f4aa406db30 msgr2=0x7f4aa41334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:06.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4aabb15640 1 -- 172.21.15.64:0/158474099 shutdown_connections 2024-09-17T15:48:06.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.535+0000 7f4aabb15640 1 -- 172.21.15.64:0/158474099 wait complete. 2024-09-17T15:48:06.583 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:06.583 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-17T15:48:06.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-17T15:48:06.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 10 monclient: get_monmap_and_config 2024-09-17T15:48:06.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 10 monclient: build_initial_monmap 2024-09-17T15:48:06.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 10 monclient: monmap: 2024-09-17T15:48:06.686 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:06.686 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.686 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:06.685505+0000 2024-09-17T15:48:06.687 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:06.685505+0000 2024-09-17T15:48:06.687 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:06.687 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.687 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:06.687 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:06.687 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:06.687 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:06.687 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:06.687 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:06.687 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:06.687 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:06.687 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:06.687 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 1 Processor -- start 2024-09-17T15:48:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 1 -- start start 2024-09-17T15:48:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 10 monclient: init 2024-09-17T15:48:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 10 monclient: _add_conns ranks=[8,3,7,0,1,5,4,6,2] 2024-09-17T15:48:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 10 monclient(hunting): picked mon.noname-e con 0x7f2c380a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 10 monclient(hunting): picked mon.noname-g con 0x7f2c38135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 10 monclient(hunting): picked mon.noname-i con 0x7f2c38135f70 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c3809d6f0 con 0x7f2c38135b50 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c3809d9d0 con 0x7f2c38135f70 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c3809dcb0 con 0x7f2c380a5520 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c3d8c6640 10 monclient(hunting): authenticate will time out at 884.360807s 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c36ffd640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f2c380a5520 0x7f2c380a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38742/0 (socket says 172.21.15.64:38742) 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.684+0000 7f2c36ffd640 1 -- 172.21.15.64:0/1785899063 learned_addr learned my addr 172.21.15.64:0/1785899063 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 -- 172.21.15.64:0/1785899063 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 96398647 0 0) 0x7f2c3809dcb0 con 0x7f2c380a5520 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient(hunting): my global_id is 5174 2024-09-17T15:48:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 -- 172.21.15.64:0/1785899063 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c20003520 con 0x7f2c380a5520 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 -- 172.21.15.64:0/1785899063 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 354413826 0 0) 0x7f2c3809d6f0 con 0x7f2c38135b50 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient(hunting): my global_id is 114276 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 -- 172.21.15.64:0/1785899063 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c3809dcb0 con 0x7f2c38135b50 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 -- 172.21.15.64:0/1785899063 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1256574034 0 0) 0x7f2c3809d9d0 con 0x7f2c38135f70 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient(hunting): my global_id is 5092 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 -- 172.21.15.64:0/1785899063 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c3809d6f0 con 0x7f2c38135f70 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 -- 172.21.15.64:0/1785899063 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1211044437 0 0) 0x7f2c20003520 con 0x7f2c380a5520 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 -- 172.21.15.64:0/1785899063 >> v1:172.21.15.100:6792/0 conn(0x7f2c38135f70 legacy=0x7f2c381415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 -- 172.21.15.64:0/1785899063 >> v1:172.21.15.100:6790/0 conn(0x7f2c38135b50 legacy=0x7f2c3813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:48:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 monclient: found mon.noname-e 2024-09-17T15:48:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:48:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 -- 172.21.15.64:0/1785899063 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c3809ccd0 con 0x7f2c380a5520 2024-09-17T15:48:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient: _finish_auth 0 2024-09-17T15:48:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:48:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 -- 172.21.15.64:0/1785899063 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2c28001bb0 con 0x7f2c380a5520 2024-09-17T15:48:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:48:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient: dump: 2024-09-17T15:48:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.700 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c3d8c6640 5 monclient: authenticate success, global_id 5174 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c3d8c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 -- 172.21.15.64:0/1785899063 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c28001dd0 con 0x7f2c380a5520 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 1 -- 172.21.15.64:0/1785899063 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2c28004f60 con 0x7f2c380a5520 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c3d8c6640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c3d8c6640 4 set_mon_vals no callback set 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c35ffb640 10 monclient: dump: 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.701 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.702 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.702 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.702 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.702 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.702 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.702 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.702 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.702 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.702 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.702 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.702 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.702 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.702 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c3d8c6640 10 monclient: shutdown 2024-09-17T15:48:06.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c3d8c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:06.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.685+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/1785899063 >> v1:172.21.15.64:6793/0 conn(0x7f2c380a5520 legacy=0x7f2c380a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.686+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/1785899063 shutdown_connections 2024-09-17T15:48:06.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.686+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/1785899063 >> 172.21.15.64:0/1785899063 conn(0x7f2c3806db30 msgr2=0x7f2c380ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:06.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.686+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/1785899063 shutdown_connections 2024-09-17T15:48:06.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.686+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/1785899063 wait complete. 2024-09-17T15:48:06.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.686+0000 7f2c3d8c6640 10 monclient: build_initial_monmap 2024-09-17T15:48:06.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.686+0000 7f2c3d8c6640 10 monclient: monmap: 2024-09-17T15:48:06.703 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:06.703 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:06.703 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:06.703 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:06.704 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:06.704 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:06.704 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:06.704 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:06.704 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:06.704 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:06.704 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:06.704 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:06.704 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:06.704 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:06.704 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:06.704 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.686+0000 7f2c3d8c6640 1 Processor -- start 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c3d8c6640 1 -- start start 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c3d8c6640 10 monclient: init 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c3d8c6640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c3d8c6640 10 monclient: _add_conns ranks=[4,0,3,1,8,7,2,5,6] 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c3d8c6640 10 monclient(hunting): picked mon.noname-c con 0x7f2c380a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c3d8c6640 10 monclient(hunting): picked mon.noname-a con 0x7f2c38135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c3d8c6640 10 monclient(hunting): picked mon.noname-g con 0x7f2c38135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c3d8c6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c380a0d40 con 0x7f2c38135b50 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c3d8c6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c38134c90 con 0x7f2c38135f70 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c3d8c6640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c381347d0 con 0x7f2c380a5520 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c3d8c6640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c3d8c6640 10 monclient(hunting): authenticate will time out at 884.363646s 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c36ffd640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f2c380a5520 0x7f2c38131a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:58884/0 (socket says 172.21.15.64:58884) 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c36ffd640 1 -- 172.21.15.64:0/2898823168 learned_addr learned my addr 172.21.15.64:0/2898823168 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:06.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4057981094 0 0) 0x7f2c381347d0 con 0x7f2c380a5520 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c1ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c1ffff640 10 monclient(hunting): my global_id is 5233 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c24003520 con 0x7f2c380a5520 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344657429 0 0) 0x7f2c3809dcb0 con 0x7f2c38135b50 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c1ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c1ffff640 10 monclient(hunting): my global_id is 5139 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.687+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c381347d0 con 0x7f2c38135b50 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4148422355 0 0) 0x7f2c38134c90 con 0x7f2c38135f70 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): my global_id is 114285 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c3809dcb0 con 0x7f2c38135f70 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 931468876 0 0) 0x7f2c381347d0 con 0x7f2c38135b50 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c38134c90 con 0x7f2c38135b50 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2c2c001ad0 con 0x7f2c38135b50 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): dump: 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.707 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1174008619 0 0) 0x7f2c3809dcb0 con 0x7f2c38135f70 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c381347d0 con 0x7f2c38135f70 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2001181865 0 0) 0x7f2c24003520 con 0x7f2c380a5520 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c3809dcb0 con 0x7f2c380a5520 2024-09-17T15:48:06.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2c30001970 con 0x7f2c38135f70 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): dump: 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.709 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2c28004ea0 con 0x7f2c380a5520 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient(hunting): dump: 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.710 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1576394151 0 0) 0x7f2c38134c90 con 0x7f2c38135b50 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 >> v1:172.21.15.64:6791/0 conn(0x7f2c380a5520 legacy=0x7f2c38131a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 >> v1:172.21.15.100:6790/0 conn(0x7f2c38135f70 legacy=0x7f2c381d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:48:06.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 monclient: found mon.a 2024-09-17T15:48:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:48:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c38134fb0 con 0x7f2c38135b50 2024-09-17T15:48:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient: _finish_auth 0 2024-09-17T15:48:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:48:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c3d8c6640 5 monclient: authenticate success, global_id 5139 2024-09-17T15:48:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c3d8c6640 10 monclient: _renew_subs 2024-09-17T15:48:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c3d8c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:48:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2c381328a0 con 0x7f2c38135b50 2024-09-17T15:48:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c3d8c6640 10 monclient: _renew_subs 2024-09-17T15:48:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c3d8c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:48:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2c3809f3f0 con 0x7f2c38135b50 2024-09-17T15:48:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c2c001cf0 con 0x7f2c38135b50 2024-09-17T15:48:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2c2c003c00 con 0x7f2c38135b50 2024-09-17T15:48:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c1ffff640 10 monclient: dump: 2024-09-17T15:48:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.688+0000 7f2c357fa640 4 set_mon_vals no callback set 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.689+0000 7f2c3d8c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.689+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2c380008d0 con 0x7f2c38142f10 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.689+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f2c2c0104e0 con 0x7f2c38135b50 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.689+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(112..112 src has 1..112) ==== 3898+0+0 (unknown 4054225456 0 0) 0x7f2c2c04d8e0 con 0x7f2c38135b50 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.689+0000 7f2c377fe640 10 monclient: get_auth_request con 0x7f2c24040d50 auth_method 0 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1773853228 0 0) 0x7f2c380008d0 con 0x7f2c38142f10 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 10 monclient(hunting): my global_id is 114269 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c38134c90 con 0x7f2c38142f10 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1296063150 0 0) 0x7f2c38134c90 con 0x7f2c38142f10 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c380008d0 con 0x7f2c38142f10 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2c28001570 con 0x7f2c38142f10 2024-09-17T15:48:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:48:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 10 monclient: dump: 2024-09-17T15:48:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.717 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.717 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.717 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.717 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.717 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.717 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3934338193 0 0) 0x7f2c380008d0 con 0x7f2c38142f10 2024-09-17T15:48:06.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.100:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c3806e500 con 0x7f2c38142f10 2024-09-17T15:48:06.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2c28002000 con 0x7f2c38142f10 2024-09-17T15:48:06.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:06.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:06.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 >> v1:172.21.15.100:6791/0 conn(0x7f2c38142f10 legacy=0x7f2c3812d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.690+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 >> v1:172.21.15.100:6791/0 conn(0x7f2c38142f10 legacy=0x7f2c3812d7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:06.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.696+0000 7f2c3d8c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:06.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.696+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2c3812e5f0 con 0x7f2c380a5040 2024-09-17T15:48:06.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 182722809 0 0) 0x7f2c3812e5f0 con 0x7f2c380a5040 2024-09-17T15:48:06.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 10 monclient(hunting): my global_id is 114278 2024-09-17T15:48:06.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c380008d0 con 0x7f2c380a5040 2024-09-17T15:48:06.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2429073428 0 0) 0x7f2c380008d0 con 0x7f2c380a5040 2024-09-17T15:48:06.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c3812e5f0 con 0x7f2c380a5040 2024-09-17T15:48:06.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2c28005ba0 con 0x7f2c380a5040 2024-09-17T15:48:06.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:48:06.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 10 monclient: dump: 2024-09-17T15:48:06.720 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.720 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.720 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.720 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 675623964 0 0) 0x7f2c3812e5f0 con 0x7f2c380a5040 2024-09-17T15:48:06.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.697+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 --> v1:172.21.15.100:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2c380a0880 con 0x7f2c380a5040 2024-09-17T15:48:06.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 147804771) 0x7f2c28007d80 con 0x7f2c380a5040 2024-09-17T15:48:06.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c1ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:06.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c1ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:06.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 >> v1:172.21.15.100:6791/0 conn(0x7f2c380a5040 legacy=0x7f2c3812dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c1ffff640 1 -- 172.21.15.64:0/2898823168 >> v1:172.21.15.100:6791/0 conn(0x7f2c380a5040 legacy=0x7f2c3812dee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:06.722 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:06.722 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:06.722 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:06.722 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:06.723 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:06.723 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 9918, 2024-09-17T15:48:06.723 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:06.723 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:06.723 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:06.723 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:06.723 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:06.723 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:06.723 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:06.723 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:06.723 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:06.723 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:06.723 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.724 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:48:06.724 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:06.724 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:06.724 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:06.724 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.724 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.724 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.724 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.724 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.724 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.724 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.724 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.724 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.725 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.725 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.725 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.725 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:06.725 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:06.725 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.725 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.725 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.725 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.725 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.725 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.725 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.725 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:06.726 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:06.727 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:06.727 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:06.727 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:06.727 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:06.727 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:06.727 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:06.727 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:06.727 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:06.727 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.727 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.727 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.727 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.727 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.728 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.728 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.728 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.728 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.728 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.728 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.728 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.728 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:06.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.728 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:06.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.728 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:06.728 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:06.730 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:06.731 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.731 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.731 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.731 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:06.731 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:06.731 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.731 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.731 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.731 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:06.731 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.732 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.732 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:06.732 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:06.732 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.732 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.732 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.732 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:06.732 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:06.732 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:06.734 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.736 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:06.737 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:06.739 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.740 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.740 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.740 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.740 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:06.740 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:06.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.740 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.740 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.740 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:06.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.740 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:06.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.741 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.741 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.741 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:06.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.741 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.741 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:06.741 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:06.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/2898823168 >> v1:172.21.15.100:6812/4210963484 conn(0x7f2c24040d50 legacy=0x7f2c24043210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c3d8c6640 10 monclient: shutdown 2024-09-17T15:48:06.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c3d8c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:06.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/2898823168 >> v1:172.21.15.64:6789/0 conn(0x7f2c38135b50 legacy=0x7f2c38132190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c377fe640 1 -- 172.21.15.64:0/2898823168 reap_dead start 2024-09-17T15:48:06.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c377fe640 1 -- 172.21.15.64:0/2898823168 reap_dead start 2024-09-17T15:48:06.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/2898823168 shutdown_connections 2024-09-17T15:48:06.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/2898823168 >> 172.21.15.64:0/2898823168 conn(0x7f2c3806db30 msgr2=0x7f2c380a7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:06.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/2898823168 shutdown_connections 2024-09-17T15:48:06.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.698+0000 7f2c3d8c6640 1 -- 172.21.15.64:0/2898823168 wait complete. 2024-09-17T15:48:06.743 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:06.744 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-17T15:48:06.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-17T15:48:06.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 10 monclient: get_monmap_and_config 2024-09-17T15:48:06.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 10 monclient: build_initial_monmap 2024-09-17T15:48:06.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 10 monclient: monmap: 2024-09-17T15:48:06.847 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:06.847 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.847 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:06.846414+0000 2024-09-17T15:48:06.848 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:06.846414+0000 2024-09-17T15:48:06.848 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:06.848 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.848 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:06.848 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:06.848 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:06.848 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:06.848 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:06.848 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:06.848 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:06.848 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:06.848 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:06.848 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 1 Processor -- start 2024-09-17T15:48:06.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 1 -- start start 2024-09-17T15:48:06.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 10 monclient: init 2024-09-17T15:48:06.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:06.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 10 monclient: _add_conns ranks=[8,1,0,7,5,3,6,4,2] 2024-09-17T15:48:06.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 10 monclient(hunting): picked mon.noname-e con 0x7fbf800a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:06.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 10 monclient(hunting): picked mon.noname-f con 0x7fbf80135b50 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:06.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 10 monclient(hunting): picked mon.noname-a con 0x7fbf80135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:06.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8009d6f0 con 0x7fbf80135f70 2024-09-17T15:48:06.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8009d9d0 con 0x7fbf80135b50 2024-09-17T15:48:06.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8009dcb0 con 0x7fbf800a5520 2024-09-17T15:48:06.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:06.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf87584640 10 monclient(hunting): authenticate will time out at 884.521876s 2024-09-17T15:48:06.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf85320640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fbf800a5520 0x7fbf800a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38744/0 (socket says 172.21.15.64:38744) 2024-09-17T15:48:06.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.845+0000 7fbf85320640 1 -- 172.21.15.64:0/867721196 learned_addr learned my addr 172.21.15.64:0/867721196 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:06.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 1 -- 172.21.15.64:0/867721196 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 523600580 0 0) 0x7fbf8009d6f0 con 0x7fbf80135f70 2024-09-17T15:48:06.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient(hunting): my global_id is 5148 2024-09-17T15:48:06.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 1 -- 172.21.15.64:0/867721196 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf6c003520 con 0x7fbf80135f70 2024-09-17T15:48:06.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 1 -- 172.21.15.64:0/867721196 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3854964647 0 0) 0x7fbf8009dcb0 con 0x7fbf800a5520 2024-09-17T15:48:06.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient(hunting): my global_id is 5183 2024-09-17T15:48:06.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 1 -- 172.21.15.64:0/867721196 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf8009d6f0 con 0x7fbf800a5520 2024-09-17T15:48:06.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 1 -- 172.21.15.64:0/867721196 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1386073608 0 0) 0x7fbf6c003520 con 0x7fbf80135f70 2024-09-17T15:48:06.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 1 -- 172.21.15.64:0/867721196 >> v1:172.21.15.64:6793/0 conn(0x7fbf800a5520 legacy=0x7fbf800a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 1 -- 172.21.15.64:0/867721196 >> v1:172.21.15.100:6789/0 conn(0x7fbf80135b50 legacy=0x7fbf8013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient: _finish_hunting 0 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 1 monclient: found mon.noname-a 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 1 -- 172.21.15.64:0/867721196 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8009ccd0 con 0x7fbf80135f70 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient: _finish_auth 0 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient: _check_auth_tickets 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 1 -- 172.21.15.64:0/867721196 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbf78001bb0 con 0x7fbf80135f70 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf87584640 5 monclient: authenticate success, global_id 5148 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf87584640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:06.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient: dump: 2024-09-17T15:48:06.860 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.860 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.860 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.860 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.860 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.860 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.860 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.860 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.860 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.860 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.860 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.860 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.861 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.861 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.861 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.861 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf87584640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:06.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 1 -- 172.21.15.64:0/867721196 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf78001dd0 con 0x7fbf80135f70 2024-09-17T15:48:06.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:06.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 1 -- 172.21.15.64:0/867721196 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbf780020c0 con 0x7fbf80135f70 2024-09-17T15:48:06.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf87584640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:06.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf87584640 4 set_mon_vals no callback set 2024-09-17T15:48:06.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:06.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.846+0000 7fbf77fff640 10 monclient: dump: 2024-09-17T15:48:06.862 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.862 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.862 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.862 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.862 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.862 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.862 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.862 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.862 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.863 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.847+0000 7fbf87584640 10 monclient: shutdown 2024-09-17T15:48:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.847+0000 7fbf87584640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.847+0000 7fbf87584640 1 -- 172.21.15.64:0/867721196 >> v1:172.21.15.64:6789/0 conn(0x7fbf80135f70 legacy=0x7fbf801415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.847+0000 7fbf87584640 1 -- 172.21.15.64:0/867721196 shutdown_connections 2024-09-17T15:48:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.847+0000 7fbf87584640 1 -- 172.21.15.64:0/867721196 >> 172.21.15.64:0/867721196 conn(0x7fbf8006db30 msgr2=0x7fbf800ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:06.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.847+0000 7fbf87584640 1 -- 172.21.15.64:0/867721196 shutdown_connections 2024-09-17T15:48:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.847+0000 7fbf87584640 1 -- 172.21.15.64:0/867721196 wait complete. 2024-09-17T15:48:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.847+0000 7fbf87584640 10 monclient: build_initial_monmap 2024-09-17T15:48:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.847+0000 7fbf87584640 10 monclient: monmap: 2024-09-17T15:48:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:06.864 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf87584640 1 Processor -- start 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf87584640 1 -- start start 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf87584640 10 monclient: init 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf87584640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf87584640 10 monclient: _add_conns ranks=[5,4,2,6,3,8,1,7,0] 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf87584640 10 monclient(hunting): picked mon.noname-h con 0x7fbf800a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf87584640 10 monclient(hunting): picked mon.noname-c con 0x7fbf80135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:06.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf87584640 10 monclient(hunting): picked mon.noname-b con 0x7fbf80135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:06.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf87584640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8009f240 con 0x7fbf80135f70 2024-09-17T15:48:06.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf87584640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf800a03b0 con 0x7fbf80135b50 2024-09-17T15:48:06.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf87584640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf800a52e0 con 0x7fbf800a5520 2024-09-17T15:48:06.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf87584640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:06.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf87584640 10 monclient(hunting): authenticate will time out at 884.524650s 2024-09-17T15:48:06.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf84b1f640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fbf80135b50 0x7fbf801d2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:58898/0 (socket says 172.21.15.64:58898) 2024-09-17T15:48:06.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf84b1f640 1 -- 172.21.15.64:0/2841091569 learned_addr learned my addr 172.21.15.64:0/2841091569 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:06.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2051997629 0 0) 0x7fbf8009f240 con 0x7fbf80135f70 2024-09-17T15:48:06.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf75ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf75ffb640 10 monclient(hunting): my global_id is 4997 2024-09-17T15:48:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.848+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf64003550 con 0x7fbf80135f70 2024-09-17T15:48:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 122648768 0 0) 0x7fbf800a52e0 con 0x7fbf800a5520 2024-09-17T15:48:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): my global_id is 114287 2024-09-17T15:48:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf8009f240 con 0x7fbf800a5520 2024-09-17T15:48:06.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3192807442 0 0) 0x7fbf800a03b0 con 0x7fbf80135b50 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): my global_id is 5242 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf800a52e0 con 0x7fbf80135b50 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1897556018 0 0) 0x7fbf64003550 con 0x7fbf80135f70 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf800a03b0 con 0x7fbf80135f70 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbf78001f80 con 0x7fbf80135f70 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): dump: 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.868 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2358115612 0 0) 0x7fbf8009f240 con 0x7fbf800a5520 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf64003550 con 0x7fbf800a5520 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbf7c001470 con 0x7fbf800a5520 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): dump: 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.870 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3894377434 0 0) 0x7fbf800a52e0 con 0x7fbf80135b50 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf8009f240 con 0x7fbf80135b50 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbf700019b0 con 0x7fbf80135b50 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:06.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient(hunting): dump: 2024-09-17T15:48:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.872 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3759951534 0 0) 0x7fbf800a03b0 con 0x7fbf80135f70 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 >> v1:172.21.15.100:6791/0 conn(0x7fbf800a5520 legacy=0x7fbf800a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 >> v1:172.21.15.64:6791/0 conn(0x7fbf80135b50 legacy=0x7fbf801d2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 monclient: found mon.b 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf800a8fc0 con 0x7fbf80135f70 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient: _finish_auth 0 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:48:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:06.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf87584640 5 monclient: authenticate success, global_id 4997 2024-09-17T15:48:06.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf87584640 10 monclient: _renew_subs 2024-09-17T15:48:06.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf87584640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:06.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf87584640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf80135320 con 0x7fbf80135f70 2024-09-17T15:48:06.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf78001420 con 0x7fbf80135f70 2024-09-17T15:48:06.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:06.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbf780042a0 con 0x7fbf80135f70 2024-09-17T15:48:06.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:06.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf75ffb640 10 monclient: dump: 2024-09-17T15:48:06.874 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.874 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.874 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.875 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.875 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.875 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.875 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.875 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.875 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.875 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.875 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.875 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.875 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.875 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.875 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.875 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf87584640 10 monclient: _renew_subs 2024-09-17T15:48:06.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf777fe640 4 set_mon_vals no callback set 2024-09-17T15:48:06.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf87584640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:48:06.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.849+0000 7fbf87584640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf800a0690 con 0x7fbf80135f70 2024-09-17T15:48:06.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.850+0000 7fbf87584640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:06.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.850+0000 7fbf87584640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbf80050ee0 con 0x7fbf80142f10 2024-09-17T15:48:06.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.850+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fbf78010b80 con 0x7fbf80135f70 2024-09-17T15:48:06.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.850+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(112..112 src has 1..112) ==== 3898+0+0 (unknown 4054225456 0 0) 0x7fbf7804dc90 con 0x7fbf80135f70 2024-09-17T15:48:06.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf84b1f640 10 monclient: get_auth_request con 0x7fbf64040cf0 auth_method 0 2024-09-17T15:48:06.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4293400215 0 0) 0x7fbf7c0047e0 con 0x7fbf80142f10 2024-09-17T15:48:06.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 10 monclient(hunting): my global_id is 5101 2024-09-17T15:48:06.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf800a03b0 con 0x7fbf80142f10 2024-09-17T15:48:06.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3034627462 0 0) 0x7fbf800a03b0 con 0x7fbf80142f10 2024-09-17T15:48:06.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf7c0047e0 con 0x7fbf80142f10 2024-09-17T15:48:06.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbf7c004fe0 con 0x7fbf80142f10 2024-09-17T15:48:06.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 10 monclient: dump: 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.878 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.879 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2911611409 0 0) 0x7fbf7c0047e0 con 0x7fbf80142f10 2024-09-17T15:48:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.100:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf800a0030 con 0x7fbf80142f10 2024-09-17T15:48:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbf7c0065e0 con 0x7fbf80142f10 2024-09-17T15:48:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 >> v1:172.21.15.100:6792/0 conn(0x7fbf80142f10 legacy=0x7fbf8012d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.851+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 >> v1:172.21.15.100:6792/0 conn(0x7fbf80142f10 legacy=0x7fbf8012d5e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:06.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.857+0000 7fbf87584640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.857+0000 7fbf87584640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbf800ab590 con 0x7fbf801cd730 2024-09-17T15:48:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.858+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 638321378 0 0) 0x7fbf800ab590 con 0x7fbf801cd730 2024-09-17T15:48:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.858+0000 7fbf75ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:48:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.858+0000 7fbf75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.858+0000 7fbf75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.858+0000 7fbf75ffb640 10 monclient(hunting): my global_id is 5110 2024-09-17T15:48:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.858+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf7c0047e0 con 0x7fbf801cd730 2024-09-17T15:48:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.858+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3409409897 0 0) 0x7fbf7c0047e0 con 0x7fbf801cd730 2024-09-17T15:48:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.858+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf800ab590 con 0x7fbf801cd730 2024-09-17T15:48:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.858+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbf7c0056a0 con 0x7fbf801cd730 2024-09-17T15:48:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.858+0000 7fbf75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:06.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.858+0000 7fbf75ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:48:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.858+0000 7fbf75ffb640 10 monclient: dump: 2024-09-17T15:48:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:06.882 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:06.883 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:06.883 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:06.883 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:06.883 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:06.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1096652759 0 0) 0x7fbf800ab590 con 0x7fbf801cd730 2024-09-17T15:48:06.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 --> v1:172.21.15.100:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbf801cdb10 con 0x7fbf801cd730 2024-09-17T15:48:06.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 228246660) 0x7fbf7c006790 con 0x7fbf801cd730 2024-09-17T15:48:06.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf75ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:06.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf75ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:06.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 >> v1:172.21.15.100:6792/0 conn(0x7fbf801cd730 legacy=0x7fbf8012dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf75ffb640 1 -- 172.21.15.64:0/2841091569 >> v1:172.21.15.100:6792/0 conn(0x7fbf801cd730 legacy=0x7fbf8012dcf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:06.884 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:06.884 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:06.884 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:06.884 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:06.884 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 16, 2024-09-17T15:48:06.884 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 143494, 2024-09-17T15:48:06.884 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:06.884 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:48:06.884 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:06.884 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:48:06.884 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:06.884 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 5, 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.885 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.886 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.886 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.886 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.886 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.886 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.886 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.886 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.886 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:06.886 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:06.886 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.886 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.886 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:06.887 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:06.888 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:06.888 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:06.888 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:06.888 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:06.888 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:06.888 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:06.888 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:06.888 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:06.888 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:06.888 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:06.888 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:06.888 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:06.888 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:06.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:06.890 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.891 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.891 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.891 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:06.891 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:06.891 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.891 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.891 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.892 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:06.892 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.892 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.892 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:06.892 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:06.892 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.892 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.892 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.893 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:06.893 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:06.893 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.893 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.893 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:06.893 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.893 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.893 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:06.893 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:06.894 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.894 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.894 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.894 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:06.894 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:06.894 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.894 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.894 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.894 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:06.894 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.895 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:06.897 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.898 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.898 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.898 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:06.898 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:06.898 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.898 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.898 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.898 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:06.898 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.899 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.899 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:06.899 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:06.899 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.899 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.899 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.899 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:06.899 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:06.899 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.899 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.900 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:06.900 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.900 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.900 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:06.900 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:06.900 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.900 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.900 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:06.901 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:06.902 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:06.902 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:06.902 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:06.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.902 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.902 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:06.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:06.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.902 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.902 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.902 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:06.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.903 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.903 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:06.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.903 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.903 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.903 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 4 2024-09-17T15:48:06.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.903 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:06.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": [ 2024-09-17T15:48:06.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:06.903 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:06.903 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:06.903 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:06.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:06.904 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:06.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:06.904 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:06.904 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:06.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf87584640 1 -- 172.21.15.64:0/2841091569 >> v1:172.21.15.100:6812/4210963484 conn(0x7fbf64040cf0 legacy=0x7fbf640431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf87584640 10 monclient: shutdown 2024-09-17T15:48:06.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf87584640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf87584640 1 -- 172.21.15.64:0/2841091569 >> v1:172.21.15.64:6790/0 conn(0x7fbf80135f70 legacy=0x7fbf801d61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf85b21640 1 -- 172.21.15.64:0/2841091569 reap_dead start 2024-09-17T15:48:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf85b21640 1 -- 172.21.15.64:0/2841091569 reap_dead start 2024-09-17T15:48:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf87584640 1 -- 172.21.15.64:0/2841091569 shutdown_connections 2024-09-17T15:48:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf87584640 1 -- 172.21.15.64:0/2841091569 >> 172.21.15.64:0/2841091569 conn(0x7fbf8006db30 msgr2=0x7fbf800a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf87584640 1 -- 172.21.15.64:0/2841091569 shutdown_connections 2024-09-17T15:48:06.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:06.859+0000 7fbf87584640 1 -- 172.21.15.64:0/2841091569 wait complete. 2024-09-17T15:48:06.906 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:48:06.907 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a', 'd', 'b', 'h'] 2024-09-17T15:48:06.907 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:48:06.907 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-17T15:48:06.907 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-17T15:48:06.907 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T15:48:06.907 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:48:06.908 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:06.908 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36566]: starting. 2024-09-17T15:48:06.908 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36566]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36412 2024-09-17T15:48:06.908 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36566]: shutting down. 2024-09-17T15:48:06.908 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:06.909 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:06.909 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:48:06.909 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:48:06.919 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T15:48:06.919 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-09-17T15:48:06.919 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-09-17T15:48:06.919 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-09-17T15:48:06.919 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:48:06.970 INFO:tasks.ceph.mon.d:Stopped 2024-09-17T15:48:06.970 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T15:48:06.970 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T15:48:06.970 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:48:06.970 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:48:07.022 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:48:07.022 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h 2024-09-17T15:48:07.022 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-09-17T15:48:07.022 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-09-17T15:48:07.022 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:48:07.032 INFO:tasks.ceph.mon.h:Stopped 2024-09-17T15:48:07.032 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-09-17T15:48:07.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:48:07.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.159+0000 7fb59aa8c640 10 monclient: get_monmap_and_config 2024-09-17T15:48:07.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.159+0000 7fb59aa8c640 10 monclient: build_initial_monmap 2024-09-17T15:48:07.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.159+0000 7fb59aa8c640 10 monclient: monmap: 2024-09-17T15:48:07.161 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:07.161 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:07.161 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:07.160105+0000 2024-09-17T15:48:07.161 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:07.160105+0000 2024-09-17T15:48:07.161 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:07.161 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:07.161 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:07.161 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:07.161 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:07.162 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:07.162 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:07.162 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:07.162 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:07.162 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:07.162 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:07.162 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:07.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.159+0000 7fb59aa8c640 1 Processor -- start 2024-09-17T15:48:07.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb59aa8c640 1 -- start start 2024-09-17T15:48:07.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb59aa8c640 10 monclient: init 2024-09-17T15:48:07.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb59aa8c640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:07.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb59aa8c640 10 monclient: _add_conns ranks=[1,3,6,7,2,5,8,4,0] 2024-09-17T15:48:07.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb59aa8c640 10 monclient(hunting): picked mon.noname-f con 0x7fb5940a54b0 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:07.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb59aa8c640 10 monclient(hunting): picked mon.noname-g con 0x7fb594135c80 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:07.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb59aa8c640 10 monclient(hunting): picked mon.noname-d con 0x7fb5941360a0 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:07.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb59aa8c640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5940aacb0 con 0x7fb5940a54b0 2024-09-17T15:48:07.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb59aa8c640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb59409d780 con 0x7fb594135c80 2024-09-17T15:48:07.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb59aa8c640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb59409da60 con 0x7fb5941360a0 2024-09-17T15:48:07.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb59aa8c640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:07.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb59aa8c640 10 monclient(hunting): authenticate will time out at 884.835577s 2024-09-17T15:48:07.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb599029640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7fb5941360a0 legacy=0x7fb594141640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:48:07.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb598828640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fb5940a54b0 0x7fb5940a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.64:49968/0 (socket says 172.21.15.64:49968) 2024-09-17T15:48:07.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.160+0000 7fb598828640 1 -- 172.21.15.64:0/1428892913 learned_addr learned my addr 172.21.15.64:0/1428892913 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:07.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 1 -- 172.21.15.64:0/1428892913 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 49758271 0 0) 0x7fb5940aacb0 con 0x7fb5940a54b0 2024-09-17T15:48:07.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:07.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:07.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:07.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient(hunting): my global_id is 5275 2024-09-17T15:48:07.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 1 -- 172.21.15.64:0/1428892913 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb584003520 con 0x7fb5940a54b0 2024-09-17T15:48:07.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 1 -- 172.21.15.64:0/1428892913 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 60388567 0 0) 0x7fb59409d780 con 0x7fb594135c80 2024-09-17T15:48:07.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:07.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:07.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:07.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient(hunting): my global_id is 114312 2024-09-17T15:48:07.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 1 -- 172.21.15.64:0/1428892913 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5940aacb0 con 0x7fb594135c80 2024-09-17T15:48:07.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 1 -- 172.21.15.64:0/1428892913 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 137271625 0 0) 0x7fb5940aacb0 con 0x7fb594135c80 2024-09-17T15:48:07.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 1 -- 172.21.15.64:0/1428892913 >> v1:172.21.15.64:6792/0 conn(0x7fb5941360a0 legacy=0x7fb594141640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:07.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 1 -- 172.21.15.64:0/1428892913 >> v1:172.21.15.100:6789/0 conn(0x7fb5940a54b0 legacy=0x7fb5940a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:07.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient: _finish_hunting 0 2024-09-17T15:48:07.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 1 monclient: found mon.noname-g 2024-09-17T15:48:07.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:07.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.100:6790/0 2024-09-17T15:48:07.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 1 -- 172.21.15.64:0/1428892913 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb59409dd40 con 0x7fb594135c80 2024-09-17T15:48:07.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient: _finish_auth 0 2024-09-17T15:48:07.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient: _check_auth_tickets 2024-09-17T15:48:07.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:07.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 1 -- 172.21.15.64:0/1428892913 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb580001810 con 0x7fb594135c80 2024-09-17T15:48:07.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:07.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:48:07.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient: dump: 2024-09-17T15:48:07.172 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:07.172 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:07.172 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:07.172 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:07.172 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:07.172 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:07.173 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:07.173 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:07.173 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:07.173 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:07.173 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:07.173 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:07.173 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:07.173 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:07.173 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:07.173 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:07.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb59aa8c640 5 monclient: authenticate success, global_id 114312 2024-09-17T15:48:07.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb59aa8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:07.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 1 -- 172.21.15.64:0/1428892913 <== mon.3 v1:172.21.15.100:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb5800019b0 con 0x7fb594135c80 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb59aa8c640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb59aa8c640 4 set_mon_vals no callback set 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 1 -- 172.21.15.64:0/1428892913 <== mon.3 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb580003950 con 0x7fb594135c80 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb5937fe640 10 monclient: dump: 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:07.174 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb59aa8c640 10 monclient: shutdown 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb59aa8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb59aa8c640 1 -- 172.21.15.64:0/1428892913 >> v1:172.21.15.100:6790/0 conn(0x7fb594135c80 legacy=0x7fb59413df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb59aa8c640 1 -- 172.21.15.64:0/1428892913 shutdown_connections 2024-09-17T15:48:07.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb59aa8c640 1 -- 172.21.15.64:0/1428892913 >> 172.21.15.64:0/1428892913 conn(0x7fb59406daa0 msgr2=0x7fb5940ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:07.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb59aa8c640 1 -- 172.21.15.64:0/1428892913 shutdown_connections 2024-09-17T15:48:07.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.161+0000 7fb59aa8c640 1 -- 172.21.15.64:0/1428892913 wait complete. 2024-09-17T15:48:07.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.162+0000 7fb59aa8c640 10 monclient: build_initial_monmap 2024-09-17T15:48:07.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.162+0000 7fb59aa8c640 10 monclient: monmap: 2024-09-17T15:48:07.176 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:07.176 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:07.176 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:07.176 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:07.176 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:07.176 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:07.176 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:07.176 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:07.176 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.162+0000 7fb59aa8c640 1 Processor -- start 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.162+0000 7fb59aa8c640 1 -- start start 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.162+0000 7fb59aa8c640 10 monclient: init 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb59aa8c640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb59aa8c640 10 monclient: _add_conns ranks=[7,4,8,1,2,6,5,0,3] 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb59aa8c640 10 monclient(hunting): picked mon.noname-i con 0x7fb5940a54b0 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:07.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb59aa8c640 10 monclient(hunting): picked mon.noname-c con 0x7fb594135c80 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:07.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb59aa8c640 10 monclient(hunting): picked mon.noname-e con 0x7fb5941360a0 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:07.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb59aa8c640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb59409e980 con 0x7fb594135c80 2024-09-17T15:48:07.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb59aa8c640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5940a0320 con 0x7fb5940a54b0 2024-09-17T15:48:07.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb59aa8c640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5941cac90 con 0x7fb5941360a0 2024-09-17T15:48:07.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb59aa8c640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:07.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb59aa8c640 10 monclient(hunting): authenticate will time out at 884.838493s 2024-09-17T15:48:07.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb593fff640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fb594135c80 0x7fb5941ca580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:58918/0 (socket says 172.21.15.64:58918) 2024-09-17T15:48:07.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb593fff640 1 -- 172.21.15.64:0/186176950 learned_addr learned my addr 172.21.15.64:0/186176950 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:07.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3133546627 0 0) 0x7fb59409e980 con 0x7fb594135c80 2024-09-17T15:48:07.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 10 monclient(hunting): _negotiate 2024-09-17T15:48:07.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:07.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:07.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 10 monclient(hunting): my global_id is 5278 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb578003550 con 0x7fb594135c80 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 57168904 0 0) 0x7fb5941cac90 con 0x7fb5941360a0 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 10 monclient(hunting): _negotiate 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 10 monclient(hunting): my global_id is 5201 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb59409e980 con 0x7fb5941360a0 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1290372097 0 0) 0x7fb5940a0320 con 0x7fb5940a54b0 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 10 monclient(hunting): _negotiate 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 10 monclient(hunting): my global_id is 5128 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.163+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5941cac90 con 0x7fb5940a54b0 2024-09-17T15:48:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 836606293 0 0) 0x7fb578003550 con 0x7fb594135c80 2024-09-17T15:48:07.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5940a0320 con 0x7fb594135c80 2024-09-17T15:48:07.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb580001390 con 0x7fb594135c80 2024-09-17T15:48:07.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:07.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:07.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient(hunting): dump: 2024-09-17T15:48:07.180 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:07.180 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:07.180 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:07.180 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:07.180 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:07.180 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:07.180 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:07.180 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2947101228 0 0) 0x7fb59409e980 con 0x7fb5941360a0 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb578003550 con 0x7fb5941360a0 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb58c0044d0 con 0x7fb5941360a0 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:07.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient(hunting): dump: 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:07.182 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:07.183 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:07.183 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:07.183 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:07.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2128237005 0 0) 0x7fb5941cac90 con 0x7fb5940a54b0 2024-09-17T15:48:07.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb59409e980 con 0x7fb5940a54b0 2024-09-17T15:48:07.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb588001590 con 0x7fb5940a54b0 2024-09-17T15:48:07.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:07.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:48:07.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient(hunting): dump: 2024-09-17T15:48:07.183 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:07.183 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:07.183 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:07.183 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:07.184 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:07.184 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:07.184 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:07.184 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:07.184 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:07.184 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:07.184 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:07.184 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:07.184 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:07.184 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:07.184 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:07.184 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:07.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3528810951 0 0) 0x7fb5940a0320 con 0x7fb594135c80 2024-09-17T15:48:07.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 >> v1:172.21.15.64:6793/0 conn(0x7fb5941360a0 legacy=0x7fb5941d5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:07.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 >> v1:172.21.15.100:6792/0 conn(0x7fb5940a54b0 legacy=0x7fb5941c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:07.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient: _finish_hunting 0 2024-09-17T15:48:07.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 monclient: found mon.c 2024-09-17T15:48:07.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:07.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:07.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5940a7380 con 0x7fb594135c80 2024-09-17T15:48:07.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient: _finish_auth 0 2024-09-17T15:48:07.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient: _check_auth_tickets 2024-09-17T15:48:07.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:07.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb59aa8c640 5 monclient: authenticate success, global_id 5278 2024-09-17T15:48:07.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb59aa8c640 10 monclient: _renew_subs 2024-09-17T15:48:07.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb59aa8c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb59aa8c640 1 -- 172.21.15.64:0/186176950 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb5940a6e50 con 0x7fb594135c80 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb59aa8c640 10 monclient: _renew_subs 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb59aa8c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb59aa8c640 1 -- 172.21.15.64:0/186176950 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fb5940a0040 con 0x7fb594135c80 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb580001880 con 0x7fb594135c80 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb580003f50 con 0x7fb594135c80 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb592ffd640 4 set_mon_vals no callback set 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.164+0000 7fb5917fa640 10 monclient: dump: 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:07.186 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:07.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.165+0000 7fb59aa8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:07.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.165+0000 7fb59aa8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:07.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.165+0000 7fb59aa8c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:07.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.165+0000 7fb59aa8c640 1 -- 172.21.15.64:0/186176950 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb574005080 con 0x7fb594135c80 2024-09-17T15:48:07.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.165+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fb580004170 con 0x7fb594135c80 2024-09-17T15:48:07.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.165+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(113..113 src has 1..113) ==== 3506+0+0 (unknown 2785139152 0 0) 0x7fb58004ea80 con 0x7fb594135c80 2024-09-17T15:48:07.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.165+0000 7fb598828640 10 monclient: get_auth_request con 0x7fb578040d80 auth_method 0 2024-09-17T15:48:07.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.170+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fb580017930 con 0x7fb594135c80 2024-09-17T15:48:07.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.170+0000 7fb5917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:07.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.170+0000 7fb5917fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:07.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.430+0000 7fb59aa8c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:48:07.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.430+0000 7fb59aa8c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:07.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.430+0000 7fb59aa8c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:07.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.430+0000 7fb59aa8c640 1 -- 172.21.15.64:0/186176950 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb574005370 con 0x7fb594135c80 2024-09-17T15:48:07.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.431+0000 7fb5917fa640 1 -- 172.21.15.64:0/186176950 <== mon.4 v1:172.21.15.64:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 1950137602) 0x7fb580017230 con 0x7fb594135c80 2024-09-17T15:48:07.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.431+0000 7fb5917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:07.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.431+0000 7fb5917fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:07.433 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":16,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:48:07.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.433+0000 7fb59aa8c640 1 -- 172.21.15.64:0/186176950 >> v1:172.21.15.100:6812/4210963484 conn(0x7fb578040d80 legacy=0x7fb578043240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:07.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.433+0000 7fb59aa8c640 10 monclient: shutdown 2024-09-17T15:48:07.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.433+0000 7fb59aa8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:07.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.433+0000 7fb59aa8c640 1 -- 172.21.15.64:0/186176950 >> v1:172.21.15.64:6791/0 conn(0x7fb594135c80 legacy=0x7fb5941ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:07.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.433+0000 7fb59aa8c640 1 -- 172.21.15.64:0/186176950 shutdown_connections 2024-09-17T15:48:07.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.433+0000 7fb59aa8c640 1 -- 172.21.15.64:0/186176950 >> 172.21.15.64:0/186176950 conn(0x7fb59406daa0 msgr2=0x7fb5940a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:07.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.433+0000 7fb59aa8c640 1 -- 172.21.15.64:0/186176950 shutdown_connections 2024-09-17T15:48:07.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:07.433+0000 7fb59aa8c640 1 -- 172.21.15.64:0/186176950 wait complete. 2024-09-17T15:48:10.447 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:48:10.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.558+0000 7fca382bf640 10 monclient: get_monmap_and_config 2024-09-17T15:48:10.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.558+0000 7fca382bf640 10 monclient: build_initial_monmap 2024-09-17T15:48:10.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.558+0000 7fca382bf640 10 monclient: monmap: 2024-09-17T15:48:10.560 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:10.560 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:10.560 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:10.559046+0000 2024-09-17T15:48:10.560 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:10.559046+0000 2024-09-17T15:48:10.560 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:10.560 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:10.560 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:10.560 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:10.561 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:10.561 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.558+0000 7fca382bf640 1 Processor -- start 2024-09-17T15:48:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.558+0000 7fca382bf640 1 -- start start 2024-09-17T15:48:10.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.558+0000 7fca382bf640 10 monclient: init 2024-09-17T15:48:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.559+0000 7fca382bf640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.559+0000 7fca382bf640 10 monclient: _add_conns ranks=[5,6,0,2,7,8,3,4,1] 2024-09-17T15:48:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.559+0000 7fca382bf640 10 monclient(hunting): picked mon.noname-h con 0x7fca300a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.559+0000 7fca382bf640 10 monclient(hunting): picked mon.noname-d con 0x7fca30135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.559+0000 7fca382bf640 10 monclient(hunting): picked mon.noname-a con 0x7fca301360a0 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.559+0000 7fca382bf640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca300aacb0 con 0x7fca301360a0 2024-09-17T15:48:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.559+0000 7fca382bf640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3009d780 con 0x7fca300a54b0 2024-09-17T15:48:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.559+0000 7fca382bf640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3009da60 con 0x7fca30135c80 2024-09-17T15:48:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.559+0000 7fca382bf640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.559+0000 7fca382bf640 10 monclient(hunting): authenticate will time out at 888.234675s 2024-09-17T15:48:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.559+0000 7fca3585a640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7fca30135c80 legacy=0x7fca3013df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:48:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.559+0000 7fca3685c640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7fca301360a0 legacy=0x7fca30141640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-09-17T15:48:10.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.559+0000 7fca3605b640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7fca300a54b0 legacy=0x7fca300a5890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:48:10.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.760+0000 7fca3585a640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7fca30135c80 legacy=0x7fca3013df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:48:10.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.760+0000 7fca3685c640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7fca301360a0 legacy=0x7fca30141640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-09-17T15:48:10.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:10.760+0000 7fca3605b640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7fca300a54b0 legacy=0x7fca300a5890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:48:11.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:11.161+0000 7fca3585a640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7fca30135c80 legacy=0x7fca3013df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:48:11.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:11.161+0000 7fca3685c640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7fca301360a0 legacy=0x7fca30141640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-09-17T15:48:11.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:11.161+0000 7fca3605b640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7fca300a54b0 legacy=0x7fca300a5890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:48:11.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:11.962+0000 7fca3605b640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7fca300a54b0 legacy=0x7fca300a5890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:48:11.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:11.962+0000 7fca3585a640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7fca30135c80 legacy=0x7fca3013df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:48:11.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:11.962+0000 7fca3685c640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7fca301360a0 legacy=0x7fca30141640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-09-17T15:48:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.559+0000 7fca23fff640 10 monclient(hunting): tick 2024-09-17T15:48:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.559+0000 7fca23fff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:48:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.559+0000 7fca23fff640 1 monclient(hunting): continuing hunt 2024-09-17T15:48:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.559+0000 7fca23fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:48:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.559+0000 7fca23fff640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7fca30135c80 legacy=0x7fca3013df10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.559+0000 7fca23fff640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7fca300a54b0 legacy=0x7fca300a5890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.559+0000 7fca23fff640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7fca301360a0 legacy=0x7fca30141640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:13.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.559+0000 7fca23fff640 10 monclient: _add_conns ranks=[2,8,4,1,3,7] 2024-09-17T15:48:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.559+0000 7fca23fff640 10 monclient(hunting): picked mon.noname-b con 0x7fca14003780 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.559+0000 7fca23fff640 10 monclient(hunting): picked mon.noname-e con 0x7fca14007410 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.560+0000 7fca23fff640 10 monclient(hunting): picked mon.noname-c con 0x7fca1400b1a0 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.560+0000 7fca23fff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3009dd40 con 0x7fca14003780 2024-09-17T15:48:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.560+0000 7fca23fff640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca300aacb0 con 0x7fca1400b1a0 2024-09-17T15:48:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.560+0000 7fca3605b640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7fca14003780 legacy=0x7fca14005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-09-17T15:48:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.560+0000 7fca23fff640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3009d780 con 0x7fca14007410 2024-09-17T15:48:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.560+0000 7fca23fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.560+0000 7fca3585a640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fca14007410 0x7fca140098d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:38754/0 (socket says 172.21.15.64:38754) 2024-09-17T15:48:13.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.560+0000 7fca3585a640 1 -- 172.21.15.64:0/2040153928 learned_addr learned my addr 172.21.15.64:0/2040153928 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:13.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:13.761+0000 7fca3605b640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.64:6790/0 conn(0x7fca14003780 legacy=0x7fca14005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-09-17T15:48:14.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:14.162+0000 7fca3605b640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.64:6790/0 conn(0x7fca14003780 legacy=0x7fca14005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-09-17T15:48:14.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:14.963+0000 7fca3605b640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.64:6790/0 conn(0x7fca14003780 legacy=0x7fca14005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-09-17T15:48:16.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 10 monclient(hunting): tick 2024-09-17T15:48:16.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:48:16.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 1 monclient(hunting): continuing hunt 2024-09-17T15:48:16.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:48:16.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.64:6793/0 conn(0x7fca14007410 legacy=0x7fca140098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:16.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.64:6791/0 conn(0x7fca1400b1a0 legacy=0x7fca1400d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:16.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.64:6790/0 conn(0x7fca14003780 legacy=0x7fca14005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:16.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca3685c640 1 -- 172.21.15.64:0/2040153928 reap_dead start 2024-09-17T15:48:16.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 10 monclient: _add_conns ranks=[7,3,1] 2024-09-17T15:48:16.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 10 monclient(hunting): picked mon.noname-i con 0x7fca14010020 addr v1:172.21.15.100:6792/0 2024-09-17T15:48:16.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca3685c640 1 -- 172.21.15.64:0/2040153928 reap_dead start 2024-09-17T15:48:16.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 10 monclient(hunting): picked mon.noname-g con 0x7fca140104a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:16.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 10 monclient(hunting): picked mon.noname-f con 0x7fca140109f0 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:16.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 1 -- 172.21.15.64:0/2040153928 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3009da60 con 0x7fca140109f0 2024-09-17T15:48:16.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 1 -- 172.21.15.64:0/2040153928 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3009dd40 con 0x7fca140104a0 2024-09-17T15:48:16.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 1 -- 172.21.15.64:0/2040153928 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca14012200 con 0x7fca14010020 2024-09-17T15:48:16.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:16.560+0000 7fca23fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 10 monclient(hunting): tick 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 1 monclient(hunting): continuing hunt 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.100:6792/0 conn(0x7fca14010020 legacy=0x7fca14005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.100:6790/0 conn(0x7fca140104a0 legacy=0x7fca14009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.100:6789/0 conn(0x7fca140109f0 legacy=0x7fca1400d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 10 monclient: _add_conns ranks=[6,8,1,0,4,5,3,2,7] 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 10 monclient(hunting): picked mon.noname-d con 0x7fca14011190 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:19.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 10 monclient(hunting): picked mon.noname-e con 0x7fca140123d0 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:19.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 10 monclient(hunting): picked mon.noname-f con 0x7fca140109f0 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:19.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 1 -- 172.21.15.64:0/2040153928 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca14013480 con 0x7fca140109f0 2024-09-17T15:48:19.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 1 -- 172.21.15.64:0/2040153928 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca14012f60 con 0x7fca14011190 2024-09-17T15:48:19.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 1 -- 172.21.15.64:0/2040153928 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca14013130 con 0x7fca140123d0 2024-09-17T15:48:19.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca23fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:19.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.560+0000 7fca3605b640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.64:6792/0 conn(0x7fca14011190 legacy=0x7fca140115d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:48:19.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:19.762+0000 7fca3605b640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.64:6792/0 conn(0x7fca14011190 legacy=0x7fca140115d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:48:20.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:20.164+0000 7fca3605b640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.64:6792/0 conn(0x7fca14011190 legacy=0x7fca140115d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:48:20.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:20.965+0000 7fca3605b640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.64:6792/0 conn(0x7fca14011190 legacy=0x7fca140115d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:48:21.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 1 -- 172.21.15.64:0/2040153928 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4008269437 0 0) 0x7fca14013130 con 0x7fca140123d0 2024-09-17T15:48:21.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient(hunting): _negotiate 2024-09-17T15:48:21.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient(hunting): my global_id is 5210 2024-09-17T15:48:21.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 1 -- 172.21.15.64:0/2040153928 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca1c003520 con 0x7fca140123d0 2024-09-17T15:48:21.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 1 -- 172.21.15.64:0/2040153928 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 288087758 0 0) 0x7fca1c003520 con 0x7fca140123d0 2024-09-17T15:48:21.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.64:6792/0 conn(0x7fca14011190 legacy=0x7fca140115d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:21.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.100:6789/0 conn(0x7fca140109f0 legacy=0x7fca1400d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient: _finish_hunting 0 2024-09-17T15:48:21.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 1 monclient: found mon.noname-e 2024-09-17T15:48:21.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:21.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:48:21.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 1 -- 172.21.15.64:0/2040153928 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1401b730 con 0x7fca140123d0 2024-09-17T15:48:21.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient: _finish_auth 0 2024-09-17T15:48:21.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient: _check_auth_tickets 2024-09-17T15:48:21.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:21.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 1 -- 172.21.15.64:0/2040153928 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fca2c004330 con 0x7fca140123d0 2024-09-17T15:48:21.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient: dump: 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:21.911 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:21.911 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:21.911 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:21.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca382bf640 5 monclient: authenticate success, global_id 5210 2024-09-17T15:48:21.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca382bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:21.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 1 -- 172.21.15.64:0/2040153928 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fca2c0044d0 con 0x7fca140123d0 2024-09-17T15:48:21.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:21.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 1 -- 172.21.15.64:0/2040153928 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fca2c004a80 con 0x7fca140123d0 2024-09-17T15:48:21.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca382bf640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:21.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca382bf640 4 set_mon_vals no callback set 2024-09-17T15:48:21.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:21.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.905+0000 7fca35059640 10 monclient: dump: 2024-09-17T15:48:21.912 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:21.912 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:21.912 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:21.912 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:21.912 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.912 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:21.912 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:21.912 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:21.912 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:21.912 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:21.912 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:21.912 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:21.913 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:21.913 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:21.913 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:21.913 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:21.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.906+0000 7fca382bf640 10 monclient: shutdown 2024-09-17T15:48:21.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.906+0000 7fca382bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:21.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.906+0000 7fca382bf640 1 -- 172.21.15.64:0/2040153928 >> v1:172.21.15.64:6793/0 conn(0x7fca140123d0 legacy=0x7fca14012850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.906+0000 7fca3685c640 1 -- 172.21.15.64:0/2040153928 reap_dead start 2024-09-17T15:48:21.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.906+0000 7fca382bf640 1 -- 172.21.15.64:0/2040153928 shutdown_connections 2024-09-17T15:48:21.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.906+0000 7fca382bf640 1 -- 172.21.15.64:0/2040153928 >> 172.21.15.64:0/2040153928 conn(0x7fca3006daa0 msgr2=0x7fca300ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:21.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.906+0000 7fca382bf640 1 -- 172.21.15.64:0/2040153928 shutdown_connections 2024-09-17T15:48:21.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.906+0000 7fca382bf640 1 -- 172.21.15.64:0/2040153928 wait complete. 2024-09-17T15:48:21.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.907+0000 7fca382bf640 10 monclient: build_initial_monmap 2024-09-17T15:48:21.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.907+0000 7fca382bf640 10 monclient: monmap: 2024-09-17T15:48:21.915 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:21.915 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:21.915 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:21.915 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:21.915 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:21.915 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:21.915 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca382bf640 1 Processor -- start 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca382bf640 1 -- start start 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca382bf640 10 monclient: init 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca382bf640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:21.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca382bf640 10 monclient: _add_conns ranks=[5,3,8,1,2,4,0,6,7] 2024-09-17T15:48:21.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca382bf640 10 monclient(hunting): picked mon.noname-h con 0x7fca140123d0 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:21.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca382bf640 10 monclient(hunting): picked mon.noname-g con 0x7fca3009e440 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:21.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca382bf640 10 monclient(hunting): picked mon.noname-e con 0x7fca301ce3c0 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:21.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca382bf640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca301cf040 con 0x7fca3009e440 2024-09-17T15:48:21.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca382bf640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca301ce870 con 0x7fca140123d0 2024-09-17T15:48:21.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca382bf640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca300a0270 con 0x7fca301ce3c0 2024-09-17T15:48:21.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca3605b640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7fca140123d0 legacy=0x7fca300a4a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:48:21.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca382bf640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:21.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.908+0000 7fca382bf640 10 monclient(hunting): authenticate will time out at 899.583928s 2024-09-17T15:48:21.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.909+0000 7fca3685c640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fca301ce3c0 0x7fca301d6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:44344/0 (socket says 172.21.15.64:44344) 2024-09-17T15:48:21.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.909+0000 7fca3685c640 1 -- 172.21.15.64:0/3875489376 learned_addr learned my addr 172.21.15.64:0/3875489376 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:21.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.916+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2827266725 0 0) 0x7fca301cf040 con 0x7fca3009e440 2024-09-17T15:48:21.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.916+0000 7fca22ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:21.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.916+0000 7fca22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.916+0000 7fca22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.916+0000 7fca22ffd640 10 monclient(hunting): my global_id is 114321 2024-09-17T15:48:21.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.916+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca14010000 con 0x7fca3009e440 2024-09-17T15:48:21.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4087267040 0 0) 0x7fca14010000 con 0x7fca3009e440 2024-09-17T15:48:21.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca301cf040 con 0x7fca3009e440 2024-09-17T15:48:21.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fca2c0049c0 con 0x7fca3009e440 2024-09-17T15:48:21.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:21.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:48:21.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient(hunting): dump: 2024-09-17T15:48:21.921 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:21.922 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:21.923 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:21.923 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:21.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:21.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 884772459 0 0) 0x7fca300a0270 con 0x7fca301ce3c0 2024-09-17T15:48:21.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:21.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient(hunting): my global_id is 5219 2024-09-17T15:48:21.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca14010000 con 0x7fca301ce3c0 2024-09-17T15:48:21.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3304493658 0 0) 0x7fca301cf040 con 0x7fca3009e440 2024-09-17T15:48:21.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 >> v1:172.21.15.64:6793/0 conn(0x7fca301ce3c0 legacy=0x7fca301d6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 >> v1:172.21.15.100:6791/0 conn(0x7fca140123d0 legacy=0x7fca300a4a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:21.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 1 monclient: found mon.g 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3009ff90 con 0x7fca3009e440 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient: _finish_auth 0 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca382bf640 5 monclient: authenticate success, global_id 114321 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca382bf640 10 monclient: _renew_subs 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca382bf640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca382bf640 1 -- 172.21.15.64:0/3875489376 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fca3009e980 con 0x7fca3009e440 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca382bf640 10 monclient: _renew_subs 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca382bf640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:21.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca382bf640 1 -- 172.21.15.64:0/3875489376 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fca301d82b0 con 0x7fca3009e440 2024-09-17T15:48:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 <== mon.3 v1:172.21.15.100:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fca2c004370 con 0x7fca3009e440 2024-09-17T15:48:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 <== mon.3 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fca2c005150 con 0x7fca3009e440 2024-09-17T15:48:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca22ffd640 10 monclient: dump: 2024-09-17T15:48:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:21.925 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:21.926 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:21.926 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:21.926 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:21.926 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:21.926 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:21.926 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:21.926 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:21.926 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:21.926 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:21.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.917+0000 7fca34858640 4 set_mon_vals no callback set 2024-09-17T15:48:21.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.918+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 <== mon.3 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fca2c011a30 con 0x7fca3009e440 2024-09-17T15:48:21.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.919+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 <== mon.3 v1:172.21.15.100:6790/0 8 ==== osd_map(113..113 src has 1..113) ==== 3506+0+0 (unknown 2785139152 0 0) 0x7fca2c04ea60 con 0x7fca3009e440 2024-09-17T15:48:21.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.919+0000 7fca3605b640 10 monclient: get_auth_request con 0x7fca140426a0 auth_method 0 2024-09-17T15:48:21.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.919+0000 7fca382bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:21.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.919+0000 7fca382bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:21.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.919+0000 7fca382bf640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:21.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.919+0000 7fca382bf640 1 -- 172.21.15.64:0/3875489376 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca18005080 con 0x7fca3009e440 2024-09-17T15:48:21.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.924+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 <== mon.3 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fca2c017710 con 0x7fca3009e440 2024-09-17T15:48:21.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.924+0000 7fca22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:21.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:21.924+0000 7fca22ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:22.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.192+0000 7fca382bf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.192+0000 7fca382bf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.192+0000 7fca382bf640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.192+0000 7fca382bf640 1 -- 172.21.15.64:0/3875489376 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fca18005370 con 0x7fca3009e440 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.192+0000 7fca22ffd640 1 -- 172.21.15.64:0/3875489376 <== mon.3 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2705 (unknown 254746657 0 1019430547) 0x7fca2c020970 con 0x7fca3009e440 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.192+0000 7fca22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.192+0000 7fca22ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":20,"quorum":[1,3,4,7,8],"quorum_names":["f","g","c","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.194+0000 7fca382bf640 1 -- 172.21.15.64:0/3875489376 >> v1:172.21.15.100:6812/4210963484 conn(0x7fca140426a0 legacy=0x7fca14044b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.194+0000 7fca382bf640 10 monclient: shutdown 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.194+0000 7fca382bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.194+0000 7fca382bf640 1 -- 172.21.15.64:0/3875489376 >> v1:172.21.15.100:6790/0 conn(0x7fca3009e440 legacy=0x7fca301d3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.194+0000 7fca382bf640 1 -- 172.21.15.64:0/3875489376 shutdown_connections 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.194+0000 7fca382bf640 1 -- 172.21.15.64:0/3875489376 >> 172.21.15.64:0/3875489376 conn(0x7fca3006daa0 msgr2=0x7fca30131770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.194+0000 7fca382bf640 1 -- 172.21.15.64:0/3875489376 shutdown_connections 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.194+0000 7fca382bf640 1 -- 172.21.15.64:0/3875489376 wait complete. 2024-09-17T15:48:22.207 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-09-17T15:48:22.207 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon c 2024-09-17T15:48:22.207 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:48:22.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.307+0000 7f6f0ed44640 10 monclient: get_monmap_and_config 2024-09-17T15:48:22.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.307+0000 7f6f0ed44640 10 monclient: build_initial_monmap 2024-09-17T15:48:22.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.307+0000 7f6f0ed44640 10 monclient: monmap: 2024-09-17T15:48:22.309 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:22.309 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.309 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:22.308383+0000 2024-09-17T15:48:22.309 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:22.308383+0000 2024-09-17T15:48:22.309 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:22.309 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.309 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:22.310 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:22.310 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:22.310 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:22.310 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:22.310 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:22.310 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:22.310 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:22.310 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:22.310 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0ed44640 1 Processor -- start 2024-09-17T15:48:22.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0ed44640 1 -- start start 2024-09-17T15:48:22.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0ed44640 10 monclient: init 2024-09-17T15:48:22.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0ed44640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:22.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0ed44640 10 monclient: _add_conns ranks=[4,8,2,3,6,7,1,5,0] 2024-09-17T15:48:22.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0ed44640 10 monclient(hunting): picked mon.noname-c con 0x7f6f080a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:22.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0ed44640 10 monclient(hunting): picked mon.noname-e con 0x7f6f08135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:22.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0ed44640 10 monclient(hunting): picked mon.noname-b con 0x7f6f08135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:22.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0ed44640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0809d6f0 con 0x7f6f08135f70 2024-09-17T15:48:22.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0ed44640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0809d9d0 con 0x7f6f080a5520 2024-09-17T15:48:22.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0ed44640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0809dcb0 con 0x7f6f08135b50 2024-09-17T15:48:22.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0ed44640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:22.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0ed44640 10 monclient(hunting): authenticate will time out at 899.983858s 2024-09-17T15:48:22.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0d2e1640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f6f08135f70 legacy=0x7f6f081415e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-09-17T15:48:22.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0cae0640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f6f080a5520 0x7f6f080a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:56812/0 (socket says 172.21.15.64:56812) 2024-09-17T15:48:22.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f07fff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f6f08135b50 0x7f6f0813deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:44352/0 (socket says 172.21.15.64:44352) 2024-09-17T15:48:22.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.308+0000 7f6f0cae0640 1 -- 172.21.15.64:0/453442889 learned_addr learned my addr 172.21.15.64:0/453442889 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:22.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 1 -- 172.21.15.64:0/453442889 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4287137967 0 0) 0x7f6f0809dcb0 con 0x7f6f08135b50 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient(hunting): my global_id is 5228 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 1 -- 172.21.15.64:0/453442889 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ef8003520 con 0x7f6f08135b50 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 1 -- 172.21.15.64:0/453442889 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3885436585 0 0) 0x7f6f0809d9d0 con 0x7f6f080a5520 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient(hunting): my global_id is 5287 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 1 -- 172.21.15.64:0/453442889 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f0809dcb0 con 0x7f6f080a5520 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 1 -- 172.21.15.64:0/453442889 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4017005621 0 0) 0x7f6ef8003520 con 0x7f6f08135b50 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 1 -- 172.21.15.64:0/453442889 >> v1:172.21.15.64:6791/0 conn(0x7f6f080a5520 legacy=0x7f6f080a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 1 -- 172.21.15.64:0/453442889 >> v1:172.21.15.64:6790/0 conn(0x7f6f08135f70 legacy=0x7f6f081415e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:22.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient: _finish_hunting 0 2024-09-17T15:48:22.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 1 monclient: found mon.noname-e 2024-09-17T15:48:22.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:22.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:48:22.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 1 -- 172.21.15.64:0/453442889 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0809ccd0 con 0x7f6f08135b50 2024-09-17T15:48:22.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient: _finish_auth 0 2024-09-17T15:48:22.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient: _check_auth_tickets 2024-09-17T15:48:22.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:22.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 1 -- 172.21.15.64:0/453442889 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6ef4001830 con 0x7f6f08135b50 2024-09-17T15:48:22.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f0ed44640 5 monclient: authenticate success, global_id 5228 2024-09-17T15:48:22.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f0ed44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:22.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient: dump: 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.321 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.322 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.322 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.324 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f0ed44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:22.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 1 -- 172.21.15.64:0/453442889 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ef4001a50 con 0x7f6f08135b50 2024-09-17T15:48:22.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:22.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 1 -- 172.21.15.64:0/453442889 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6ef4001d40 con 0x7f6f08135b50 2024-09-17T15:48:22.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f0ed44640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:22.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f0ed44640 4 set_mon_vals no callback set 2024-09-17T15:48:22.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.309+0000 7f6f077fe640 10 monclient: dump: 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.325 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.326 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.326 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.326 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.326 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.310+0000 7f6f0ed44640 10 monclient: shutdown 2024-09-17T15:48:22.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.310+0000 7f6f0ed44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.310+0000 7f6f0ed44640 1 -- 172.21.15.64:0/453442889 >> v1:172.21.15.64:6793/0 conn(0x7f6f08135b50 legacy=0x7f6f0813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.310+0000 7f6f0ed44640 1 -- 172.21.15.64:0/453442889 shutdown_connections 2024-09-17T15:48:22.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.310+0000 7f6f0ed44640 1 -- 172.21.15.64:0/453442889 >> 172.21.15.64:0/453442889 conn(0x7f6f0806db30 msgr2=0x7f6f080ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.310+0000 7f6f0ed44640 1 -- 172.21.15.64:0/453442889 shutdown_connections 2024-09-17T15:48:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.310+0000 7f6f0ed44640 1 -- 172.21.15.64:0/453442889 wait complete. 2024-09-17T15:48:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.310+0000 7f6f0ed44640 10 monclient: build_initial_monmap 2024-09-17T15:48:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.310+0000 7f6f0ed44640 10 monclient: monmap: 2024-09-17T15:48:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.310+0000 7f6f0ed44640 1 Processor -- start 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0ed44640 1 -- start start 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0ed44640 10 monclient: init 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0ed44640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0ed44640 10 monclient: _add_conns ranks=[8,4,6,7,5,2,1,3,0] 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0ed44640 10 monclient(hunting): picked mon.noname-e con 0x7f6f080a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0ed44640 10 monclient(hunting): picked mon.noname-c con 0x7f6f08135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:22.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0ed44640 10 monclient(hunting): picked mon.noname-d con 0x7f6f08135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0ed44640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0809f240 con 0x7f6f08135b50 2024-09-17T15:48:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0ed44640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f080a8dd0 con 0x7f6f08135f70 2024-09-17T15:48:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0d2e1640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7f6f08135f70 legacy=0x7f6f081d6260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:48:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0ed44640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f080a03b0 con 0x7f6f080a5520 2024-09-17T15:48:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0ed44640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0ed44640 10 monclient(hunting): authenticate will time out at 899.986715s 2024-09-17T15:48:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f07fff640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f6f08135b50 0x7f6f081d2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:56818/0 (socket says 172.21.15.64:56818) 2024-09-17T15:48:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f0cae0640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f6f080a5520 0x7f6f080a4bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:44358/0 (socket says 172.21.15.64:44358) 2024-09-17T15:48:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f07fff640 1 -- 172.21.15.64:0/1468531938 learned_addr learned my addr 172.21.15.64:0/1468531938 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2131144335 0 0) 0x7f6f0809f240 con 0x7f6f08135b50 2024-09-17T15:48:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f057fa640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f057fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.311+0000 7f6f057fa640 10 monclient(hunting): my global_id is 5296 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6eec003520 con 0x7f6f08135b50 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4222212652 0 0) 0x7f6f080a03b0 con 0x7f6f080a5520 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient(hunting): my global_id is 5237 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f0809f240 con 0x7f6f080a5520 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 797471250 0 0) 0x7f6eec003520 con 0x7f6f08135b50 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f080a03b0 con 0x7f6f08135b50 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6ef40017f0 con 0x7f6f08135b50 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:22.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient(hunting): dump: 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.331 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 426927790 0 0) 0x7f6f0809f240 con 0x7f6f080a5520 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6eec003520 con 0x7f6f080a5520 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6efc001550 con 0x7f6f080a5520 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient(hunting): dump: 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.332 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3471435375 0 0) 0x7f6f080a03b0 con 0x7f6f08135b50 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 >> v1:172.21.15.64:6793/0 conn(0x7f6f080a5520 legacy=0x7f6f080a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 >> v1:172.21.15.64:6792/0 conn(0x7f6f08135f70 legacy=0x7f6f081d6260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient: _finish_hunting 0 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 monclient: found mon.c 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f080a5300 con 0x7f6f08135b50 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient: _finish_auth 0 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient: _check_auth_tickets 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f0ed44640 5 monclient: authenticate success, global_id 5296 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f0ed44640 10 monclient: _renew_subs 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f0ed44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f0ed44640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6f08135040 con 0x7f6f08135b50 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f0ed44640 10 monclient: _renew_subs 2024-09-17T15:48:22.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f0ed44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:22.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f0ed44640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f6f080a0690 con 0x7f6f08135b50 2024-09-17T15:48:22.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ef4004ef0 con 0x7f6f08135b50 2024-09-17T15:48:22.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:22.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6ef40053e0 con 0x7f6f08135b50 2024-09-17T15:48:22.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f06ffd640 4 set_mon_vals no callback set 2024-09-17T15:48:22.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:22.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.312+0000 7f6f057fa640 10 monclient: dump: 2024-09-17T15:48:22.335 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.335 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.335 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.335 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.335 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.313+0000 7f6f0ed44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.313+0000 7f6f0ed44640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6ee80065c0 con 0x7f6ee8002860 2024-09-17T15:48:22.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.313+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f6ef4011cc0 con 0x7f6f08135b50 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.313+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(114..114 src has 1..114) ==== 3898+0+0 (unknown 642482365 0 0) 0x7f6ef404ec80 con 0x7f6f08135b50 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 248398144 0 0) 0x7f6ee80065c0 con 0x7f6ee8002860 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 10 monclient(hunting): my global_id is 5305 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f080a03b0 con 0x7f6ee8002860 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3043122231 0 0) 0x7f6f080a03b0 con 0x7f6ee8002860 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ee80065c0 con 0x7f6ee8002860 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f0d2e1640 10 monclient: get_auth_request con 0x7f6eec040840 auth_method 0 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6efc004b20 con 0x7f6ee8002860 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 10 monclient: dump: 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.339 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3683307854 0 0) 0x7f6ee80065c0 con 0x7f6ee8002860 2024-09-17T15:48:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ee800a160 con 0x7f6ee8002860 2024-09-17T15:48:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6efc006080 con 0x7f6ee8002860 2024-09-17T15:48:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 >> v1:172.21.15.64:6791/0 conn(0x7f6ee8002860 legacy=0x7f6ee8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.314+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 >> v1:172.21.15.64:6791/0 conn(0x7f6ee8002860 legacy=0x7f6ee8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.320+0000 7f6f0ed44640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.320+0000 7f6f0ed44640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6ee8009e50 con 0x7f6ee8009110 2024-09-17T15:48:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460613445 0 0) 0x7f6ee8009e50 con 0x7f6ee8009110 2024-09-17T15:48:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 10 monclient(hunting): my global_id is 5314 2024-09-17T15:48:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ee80065c0 con 0x7f6ee8009110 2024-09-17T15:48:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3450943853 0 0) 0x7f6ee80065c0 con 0x7f6ee8009110 2024-09-17T15:48:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ee8009e50 con 0x7f6ee8009110 2024-09-17T15:48:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6efc0068a0 con 0x7f6ee8009110 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 10 monclient: dump: 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.343 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.343 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.343 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.343 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.343 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2845258461 0 0) 0x7f6ee8009e50 con 0x7f6ee8009110 2024-09-17T15:48:22.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 --> v1:172.21.15.64:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6ee800c3a0 con 0x7f6ee8009110 2024-09-17T15:48:22.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7667 (unknown 0 0 773271660) 0x7f6efc0069d0 con 0x7f6ee8009110 2024-09-17T15:48:22.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:22.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:22.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 >> v1:172.21.15.64:6791/0 conn(0x7f6ee8009110 legacy=0x7f6ee80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.321+0000 7f6f057fa640 1 -- 172.21.15.64:0/1468531938 >> v1:172.21.15.64:6791/0 conn(0x7f6ee8009110 legacy=0x7f6ee80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:22.344 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:22.344 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:22.344 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:22.344 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:22.344 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 20, 2024-09-17T15:48:22.344 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 158978, 2024-09-17T15:48:22.344 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:22.344 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:22.344 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:22.344 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:22.344 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.345 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.346 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.347 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.347 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.347 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.347 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.347 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:22.347 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:22.347 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:22.347 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:22.347 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:22.347 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:22.347 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:22.347 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.348 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.349 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.350 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.350 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:22.350 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.350 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.350 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.350 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:22.350 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:22.350 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.350 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.350 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.352 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.352 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.352 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:22.352 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:22.352 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.352 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.352 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.352 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:22.352 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.352 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.354 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.354 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:22.354 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.354 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.354 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:22.354 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:22.354 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.354 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.354 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.355 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:22.355 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:22.355 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.355 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.355 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.355 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:22.355 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.355 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.355 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.355 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:22.356 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.357 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.357 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.357 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:22.357 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:22.357 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.357 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.357 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.357 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:22.357 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:22.357 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.359 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:22.359 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:22.359 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.359 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.359 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.359 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:22.359 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.359 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.359 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:22.360 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:22.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.362 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.362 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.362 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:22.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.362 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.362 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:48:22.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.362 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.362 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.362 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:48:22.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.362 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:22.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.363 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.363 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.363 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:48:22.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.363 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.363 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:22.363 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:22.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.322+0000 7f6f0ed44640 1 -- 172.21.15.64:0/1468531938 >> v1:172.21.15.100:6812/4210963484 conn(0x7f6eec040840 legacy=0x7f6eec042d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.322+0000 7f6f0ed44640 10 monclient: shutdown 2024-09-17T15:48:22.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.322+0000 7f6f0ed44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.322+0000 7f6f0ed44640 1 -- 172.21.15.64:0/1468531938 >> v1:172.21.15.64:6791/0 conn(0x7f6f08135b50 legacy=0x7f6f081d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.322+0000 7f6f0d2e1640 1 -- 172.21.15.64:0/1468531938 reap_dead start 2024-09-17T15:48:22.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.322+0000 7f6f0d2e1640 1 -- 172.21.15.64:0/1468531938 reap_dead start 2024-09-17T15:48:22.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.322+0000 7f6f0ed44640 1 -- 172.21.15.64:0/1468531938 shutdown_connections 2024-09-17T15:48:22.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.322+0000 7f6f0ed44640 1 -- 172.21.15.64:0/1468531938 >> 172.21.15.64:0/1468531938 conn(0x7f6f0806db30 msgr2=0x7f6f080a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.322+0000 7f6f0ed44640 1 -- 172.21.15.64:0/1468531938 shutdown_connections 2024-09-17T15:48:22.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.322+0000 7f6f0ed44640 1 -- 172.21.15.64:0/1468531938 wait complete. 2024-09-17T15:48:22.365 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:48:22.366 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon e 2024-09-17T15:48:22.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-17T15:48:22.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.465+0000 7f1263df5640 10 monclient: get_monmap_and_config 2024-09-17T15:48:22.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.465+0000 7f1263df5640 10 monclient: build_initial_monmap 2024-09-17T15:48:22.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.466+0000 7f1263df5640 10 monclient: monmap: 2024-09-17T15:48:22.467 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:22.468 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.468 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:22.466839+0000 2024-09-17T15:48:22.468 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:22.466839+0000 2024-09-17T15:48:22.468 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:22.468 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.468 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:22.468 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:22.468 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:22.468 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:22.468 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:22.468 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:22.468 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:22.468 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:22.469 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:22.469 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.466+0000 7f1263df5640 1 Processor -- start 2024-09-17T15:48:22.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.466+0000 7f1263df5640 1 -- start start 2024-09-17T15:48:22.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.466+0000 7f1263df5640 10 monclient: init 2024-09-17T15:48:22.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1263df5640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:22.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1263df5640 10 monclient: _add_conns ranks=[3,2,4,7,8,0,1,5,6] 2024-09-17T15:48:22.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1263df5640 10 monclient(hunting): picked mon.noname-g con 0x7f125c0a54b0 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:22.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1263df5640 10 monclient(hunting): picked mon.noname-b con 0x7f125c135c80 addr v1:172.21.15.64:6790/0 2024-09-17T15:48:22.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1263df5640 10 monclient(hunting): picked mon.noname-c con 0x7f125c1360a0 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:22.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1263df5640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f125c0aacb0 con 0x7f125c135c80 2024-09-17T15:48:22.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1263df5640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f125c09d780 con 0x7f125c0a54b0 2024-09-17T15:48:22.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1263df5640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f125c09da60 con 0x7f125c1360a0 2024-09-17T15:48:22.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1263df5640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:22.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1263df5640 10 monclient(hunting): authenticate will time out at 900.142324s 2024-09-17T15:48:22.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1261390640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f125c135c80 legacy=0x7f125c13df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-09-17T15:48:22.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1262392640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f125c1360a0 0x7f125c141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:56840/0 (socket says 172.21.15.64:56840) 2024-09-17T15:48:22.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1262392640 1 -- 172.21.15.64:0/3106363808 learned_addr learned my addr 172.21.15.64:0/3106363808 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:22.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 1 -- 172.21.15.64:0/3106363808 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3956964131 0 0) 0x7f125c09da60 con 0x7f125c1360a0 2024-09-17T15:48:22.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 10 monclient(hunting): my global_id is 5323 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 1 -- 172.21.15.64:0/3106363808 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1248003520 con 0x7f125c1360a0 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 1 -- 172.21.15.64:0/3106363808 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1450232072 0 0) 0x7f1248003520 con 0x7f125c1360a0 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 1 -- 172.21.15.64:0/3106363808 >> v1:172.21.15.100:6790/0 conn(0x7f125c0a54b0 legacy=0x7f125c0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 1 -- 172.21.15.64:0/3106363808 >> v1:172.21.15.64:6790/0 conn(0x7f125c135c80 legacy=0x7f125c13df10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 10 monclient: _finish_hunting 0 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 1 monclient: found mon.noname-c 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 1 -- 172.21.15.64:0/3106363808 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125c09dd40 con 0x7f125c1360a0 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 10 monclient: _finish_auth 0 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 10 monclient: _check_auth_tickets 2024-09-17T15:48:22.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:22.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 1 -- 172.21.15.64:0/3106363808 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1254001bb0 con 0x7f125c1360a0 2024-09-17T15:48:22.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1263df5640 5 monclient: authenticate success, global_id 5323 2024-09-17T15:48:22.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1263df5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:22.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:22.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.467+0000 7f1260b8f640 10 monclient: dump: 2024-09-17T15:48:22.479 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.479 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.479 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.479 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.479 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.479 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.479 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1263df5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1260b8f640 1 -- 172.21.15.64:0/3106363808 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1254001dd0 con 0x7f125c1360a0 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1260b8f640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1263df5640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1260b8f640 1 -- 172.21.15.64:0/3106363808 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1254004f60 con 0x7f125c1360a0 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1263df5640 4 set_mon_vals no callback set 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1260b8f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1260b8f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1260b8f640 10 monclient: dump: 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.482 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1263df5640 10 monclient: shutdown 2024-09-17T15:48:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1263df5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1263df5640 1 -- 172.21.15.64:0/3106363808 >> v1:172.21.15.64:6791/0 conn(0x7f125c1360a0 legacy=0x7f125c141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1263df5640 1 -- 172.21.15.64:0/3106363808 shutdown_connections 2024-09-17T15:48:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1263df5640 1 -- 172.21.15.64:0/3106363808 >> 172.21.15.64:0/3106363808 conn(0x7f125c06daa0 msgr2=0x7f125c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1263df5640 1 -- 172.21.15.64:0/3106363808 shutdown_connections 2024-09-17T15:48:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.468+0000 7f1263df5640 1 -- 172.21.15.64:0/3106363808 wait complete. 2024-09-17T15:48:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.469+0000 7f1263df5640 10 monclient: build_initial_monmap 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.469+0000 7f1263df5640 10 monclient: monmap: 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:22.484 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.469+0000 7f1263df5640 1 Processor -- start 2024-09-17T15:48:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.469+0000 7f1263df5640 1 -- start start 2024-09-17T15:48:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.469+0000 7f1263df5640 10 monclient: init 2024-09-17T15:48:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.469+0000 7f1263df5640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.469+0000 7f1263df5640 10 monclient: _add_conns ranks=[5,3,6,8,7,2,0,4,1] 2024-09-17T15:48:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.469+0000 7f1263df5640 10 monclient(hunting): picked mon.noname-h con 0x7f125c0a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.469+0000 7f1263df5640 10 monclient(hunting): picked mon.noname-g con 0x7f125c135c80 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.469+0000 7f1263df5640 10 monclient(hunting): picked mon.noname-d con 0x7f125c1360a0 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f1263df5640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f125c0a39e0 con 0x7f125c135c80 2024-09-17T15:48:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f1263df5640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f125c142630 con 0x7f125c0a54b0 2024-09-17T15:48:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f1263df5640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f125c0a3cc0 con 0x7f125c1360a0 2024-09-17T15:48:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f1263df5640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f1263df5640 10 monclient(hunting): authenticate will time out at 900.145032s 2024-09-17T15:48:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f1262392640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7f125c1360a0 legacy=0x7f125c1d62b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:48:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f1261b91640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7f125c0a54b0 legacy=0x7f125c141810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:48:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f1261390640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f125c135c80 0x7f125c141f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:46728/0 (socket says 172.21.15.64:46728) 2024-09-17T15:48:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f1261390640 1 -- 172.21.15.64:0/1523839548 learned_addr learned my addr 172.21.15.64:0/1523839548 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3989718283 0 0) 0x7f125c0a39e0 con 0x7f125c135c80 2024-09-17T15:48:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 10 monclient(hunting): my global_id is 114339 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1240003520 con 0x7f125c135c80 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 417961234 0 0) 0x7f1240003520 con 0x7f125c135c80 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f125c0a39e0 con 0x7f125c135c80 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f124c004370 con 0x7f125c135c80 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 10 monclient(hunting): dump: 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 387209598 0 0) 0x7f125c0a39e0 con 0x7f125c135c80 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 >> v1:172.21.15.64:6792/0 conn(0x7f125c1360a0 legacy=0x7f125c1d62b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 >> v1:172.21.15.100:6791/0 conn(0x7f125c0a54b0 legacy=0x7f125c141810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:22.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 1 monclient: found mon.g 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125c0a6bc0 con 0x7f125c135c80 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 10 monclient: _finish_auth 0 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 10 monclient: _check_auth_tickets 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f12527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f1263df5640 5 monclient: authenticate success, global_id 114339 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f1263df5640 10 monclient: _renew_subs 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f1263df5640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.470+0000 7f1263df5640 1 -- 172.21.15.64:0/1523839548 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f125c0a70f0 con 0x7f125c135c80 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.471+0000 7f1263df5640 10 monclient: _renew_subs 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.471+0000 7f1263df5640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:22.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.471+0000 7f1263df5640 1 -- 172.21.15.64:0/1523839548 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f125c0a3700 con 0x7f125c135c80 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.471+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.3 v1:172.21.15.100:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f124c004590 con 0x7f125c135c80 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.471+0000 7f12527fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.471+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.3 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f124c004b40 con 0x7f125c135c80 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.471+0000 7f12527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.471+0000 7f12527fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.471+0000 7f1253fff640 4 set_mon_vals no callback set 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.471+0000 7f12527fc640 10 monclient: dump: 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.489 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.490 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.490 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.490 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.490 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.490 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.490 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.490 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.490 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.471+0000 7f1263df5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.471+0000 7f1263df5640 1 -- 172.21.15.64:0/1523839548 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f12440065c0 con 0x7f1244002860 2024-09-17T15:48:22.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.471+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.3 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f124c004d60 con 0x7f125c135c80 2024-09-17T15:48:22.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.3 v1:172.21.15.100:6790/0 8 ==== osd_map(114..114 src has 1..114) ==== 3898+0+0 (unknown 642482365 0 0) 0x7f124c04edb0 con 0x7f125c135c80 2024-09-17T15:48:22.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3059851385 0 0) 0x7f12440065c0 con 0x7f1244002860 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 10 monclient(hunting): my global_id is 5246 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f125c0a39e0 con 0x7f1244002860 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2181501855 0 0) 0x7f125c0a39e0 con 0x7f1244002860 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12440065c0 con 0x7f1244002860 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1258001900 con 0x7f1244002860 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 10 monclient: dump: 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.491 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.492 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.492 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.492 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.492 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.492 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.492 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.492 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.492 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.492 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.492 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.492 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.492 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.492 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f1262392640 10 monclient: get_auth_request con 0x7f12400400f0 auth_method 0 2024-09-17T15:48:22.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 559689283 0 0) 0x7f12440065c0 con 0x7f1244002860 2024-09-17T15:48:22.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.472+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 --> v1:172.21.15.64:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f124400a160 con 0x7f1244002860 2024-09-17T15:48:22.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.473+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1258001cb0 con 0x7f1244002860 2024-09-17T15:48:22.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.473+0000 7f12527fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.473+0000 7f12527fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:22.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.473+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 >> v1:172.21.15.64:6793/0 conn(0x7f1244002860 legacy=0x7f1244004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.473+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 >> v1:172.21.15.64:6793/0 conn(0x7f1244002860 legacy=0x7f1244004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:22.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f1263df5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:22.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f1263df5640 1 -- 172.21.15.64:0/1523839548 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1244009e50 con 0x7f1244009110 2024-09-17T15:48:22.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 830857208 0 0) 0x7f1244009e50 con 0x7f1244009110 2024-09-17T15:48:22.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 10 monclient(hunting): my global_id is 5255 2024-09-17T15:48:22.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12440065c0 con 0x7f1244009110 2024-09-17T15:48:22.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1481271032 0 0) 0x7f12440065c0 con 0x7f1244009110 2024-09-17T15:48:22.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1244009e50 con 0x7f1244009110 2024-09-17T15:48:22.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1258004d90 con 0x7f1244009110 2024-09-17T15:48:22.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 10 monclient: dump: 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.497 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.497 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.497 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.497 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1067387768 0 0) 0x7f1244009e50 con 0x7f1244009110 2024-09-17T15:48:22.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.479+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 --> v1:172.21.15.64:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f124400c3f0 con 0x7f1244009110 2024-09-17T15:48:22.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7667 (unknown 0 0 443258920) 0x7f1258006190 con 0x7f1244009110 2024-09-17T15:48:22.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f12527fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:22.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f12527fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:22.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 >> v1:172.21.15.64:6793/0 conn(0x7f1244009110 legacy=0x7f12440095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f12527fc640 1 -- 172.21.15.64:0/1523839548 >> v1:172.21.15.64:6793/0 conn(0x7f1244009110 legacy=0x7f12440095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:22.498 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:22.498 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:22.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:22.498 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:22.498 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 20, 2024-09-17T15:48:22.498 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 159126, 2024-09-17T15:48:22.498 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:22.498 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:22.498 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:22.498 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:22.498 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:22.498 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:22.498 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.499 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:48:22.499 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:22.499 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:22.499 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:22.499 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.499 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.499 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.499 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.499 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.499 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.499 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.499 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.499 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.500 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.501 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.501 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.501 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.501 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:22.501 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:22.501 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:22.501 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:22.501 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:22.501 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:22.501 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:22.501 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:22.501 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:22.502 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:22.502 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:22.502 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:22.502 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:22.502 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:22.502 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:22.502 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:22.502 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:22.502 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.502 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.502 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.502 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.502 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:22.503 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.504 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.504 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.504 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:22.504 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.504 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.504 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:22.504 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:22.504 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.504 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.504 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:22.505 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:22.505 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.505 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.505 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.505 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:22.505 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.505 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.505 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.506 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:22.506 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:22.506 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.506 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.506 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:22.506 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:22.506 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.506 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.506 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:22.507 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:22.508 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.508 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.508 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.508 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:22.508 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.508 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.508 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.508 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:22.508 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:22.508 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.508 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:22.511 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.512 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.512 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:22.512 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.512 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.512 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:22.512 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:22.512 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.512 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.512 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.513 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:22.514 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.517 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.517 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.517 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:48:22.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.517 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.517 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:22.517 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:22.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f1263df5640 1 -- 172.21.15.64:0/1523839548 >> v1:172.21.15.100:6812/4210963484 conn(0x7f12400400f0 legacy=0x7f12400425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f1263df5640 10 monclient: shutdown 2024-09-17T15:48:22.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f1263df5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f1263df5640 1 -- 172.21.15.64:0/1523839548 >> v1:172.21.15.100:6790/0 conn(0x7f125c135c80 legacy=0x7f125c141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f1262392640 1 -- 172.21.15.64:0/1523839548 reap_dead start 2024-09-17T15:48:22.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f1262392640 1 -- 172.21.15.64:0/1523839548 reap_dead start 2024-09-17T15:48:22.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f1263df5640 1 -- 172.21.15.64:0/1523839548 shutdown_connections 2024-09-17T15:48:22.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f1263df5640 1 -- 172.21.15.64:0/1523839548 >> 172.21.15.64:0/1523839548 conn(0x7f125c06daa0 msgr2=0x7f125c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f1263df5640 1 -- 172.21.15.64:0/1523839548 shutdown_connections 2024-09-17T15:48:22.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.480+0000 7f1263df5640 1 -- 172.21.15.64:0/1523839548 wait complete. 2024-09-17T15:48:22.519 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:48:22.519 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon f 2024-09-17T15:48:22.520 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-17T15:48:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.620+0000 7f3f5e56b640 10 monclient: get_monmap_and_config 2024-09-17T15:48:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.620+0000 7f3f5e56b640 10 monclient: build_initial_monmap 2024-09-17T15:48:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.620+0000 7f3f5e56b640 10 monclient: monmap: 2024-09-17T15:48:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:22.621384+0000 2024-09-17T15:48:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:22.621384+0000 2024-09-17T15:48:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:22.623 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:22.623 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:22.623 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:22.623 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:22.623 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:22.623 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:22.623 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:22.623 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:22.623 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5e56b640 1 Processor -- start 2024-09-17T15:48:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5e56b640 1 -- start start 2024-09-17T15:48:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5e56b640 10 monclient: init 2024-09-17T15:48:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5e56b640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5e56b640 10 monclient: _add_conns ranks=[8,4,0,1,3,7,2,6,5] 2024-09-17T15:48:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5e56b640 10 monclient(hunting): picked mon.noname-e con 0x7f3f580a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5e56b640 10 monclient(hunting): picked mon.noname-c con 0x7f3f58135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5e56b640 10 monclient(hunting): picked mon.noname-a con 0x7f3f58135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5e56b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f5809d6f0 con 0x7f3f58135f70 2024-09-17T15:48:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5e56b640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f5809d9d0 con 0x7f3f58135b50 2024-09-17T15:48:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5e56b640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f5809dcb0 con 0x7f3f580a5520 2024-09-17T15:48:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5e56b640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5e56b640 10 monclient(hunting): authenticate will time out at 900.296902s 2024-09-17T15:48:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.621+0000 7f3f5cb08640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f3f58135f70 legacy=0x7f3f581415e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-09-17T15:48:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f577fe640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f3f58135b50 0x7f3f5813deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:56846/0 (socket says 172.21.15.64:56846) 2024-09-17T15:48:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f57fff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f3f580a5520 0x7f3f580a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:44386/0 (socket says 172.21.15.64:44386) 2024-09-17T15:48:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f577fe640 1 -- 172.21.15.64:0/1941592929 learned_addr learned my addr 172.21.15.64:0/1941592929 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 1 -- 172.21.15.64:0/1941592929 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1374162072 0 0) 0x7f3f5809dcb0 con 0x7f3f580a5520 2024-09-17T15:48:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient(hunting): my global_id is 5264 2024-09-17T15:48:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 1 -- 172.21.15.64:0/1941592929 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f40003520 con 0x7f3f580a5520 2024-09-17T15:48:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 1 -- 172.21.15.64:0/1941592929 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 215272531 0 0) 0x7f3f5809d9d0 con 0x7f3f58135b50 2024-09-17T15:48:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient(hunting): my global_id is 5332 2024-09-17T15:48:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 1 -- 172.21.15.64:0/1941592929 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f5809dcb0 con 0x7f3f58135b50 2024-09-17T15:48:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 1 -- 172.21.15.64:0/1941592929 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3149368537 0 0) 0x7f3f40003520 con 0x7f3f580a5520 2024-09-17T15:48:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 1 -- 172.21.15.64:0/1941592929 >> v1:172.21.15.64:6791/0 conn(0x7f3f58135b50 legacy=0x7f3f5813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 1 -- 172.21.15.64:0/1941592929 >> v1:172.21.15.64:6789/0 conn(0x7f3f58135f70 legacy=0x7f3f581415e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:48:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 1 monclient: found mon.noname-e 2024-09-17T15:48:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:48:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 1 -- 172.21.15.64:0/1941592929 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f5809ccd0 con 0x7f3f580a5520 2024-09-17T15:48:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient: _finish_auth 0 2024-09-17T15:48:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 1 -- 172.21.15.64:0/1941592929 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3f48001760 con 0x7f3f580a5520 2024-09-17T15:48:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f5e56b640 5 monclient: authenticate success, global_id 5264 2024-09-17T15:48:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f5e56b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:48:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient: dump: 2024-09-17T15:48:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f5e56b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 1 -- 172.21.15.64:0/1941592929 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f48001980 con 0x7f3f580a5520 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 1 -- 172.21.15.64:0/1941592929 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3f48004b20 con 0x7f3f580a5520 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f56ffd640 10 monclient: dump: 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.636 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.637 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.637 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.637 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.637 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.637 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.637 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.637 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.637 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f5e56b640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:22.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.622+0000 7f3f5e56b640 4 set_mon_vals no callback set 2024-09-17T15:48:22.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.623+0000 7f3f5e56b640 10 monclient: shutdown 2024-09-17T15:48:22.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.623+0000 7f3f5e56b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.623+0000 7f3f5e56b640 1 -- 172.21.15.64:0/1941592929 >> v1:172.21.15.64:6793/0 conn(0x7f3f580a5520 legacy=0x7f3f580a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.623+0000 7f3f5e56b640 1 -- 172.21.15.64:0/1941592929 shutdown_connections 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.623+0000 7f3f5e56b640 1 -- 172.21.15.64:0/1941592929 >> 172.21.15.64:0/1941592929 conn(0x7f3f5806db30 msgr2=0x7f3f580ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.623+0000 7f3f5e56b640 1 -- 172.21.15.64:0/1941592929 shutdown_connections 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.623+0000 7f3f5e56b640 1 -- 172.21.15.64:0/1941592929 wait complete. 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.623+0000 7f3f5e56b640 10 monclient: build_initial_monmap 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.623+0000 7f3f5e56b640 10 monclient: monmap: 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:22.638 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5e56b640 1 Processor -- start 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5e56b640 1 -- start start 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5e56b640 10 monclient: init 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5e56b640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5e56b640 10 monclient: _add_conns ranks=[1,0,4,3,7,2,8,5,6] 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5e56b640 10 monclient(hunting): picked mon.noname-f con 0x7f3f580a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:22.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5e56b640 10 monclient(hunting): picked mon.noname-a con 0x7f3f58135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5e56b640 10 monclient(hunting): picked mon.noname-c con 0x7f3f58135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:48:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5e56b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f580a04b0 con 0x7f3f58135b50 2024-09-17T15:48:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5e56b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f58134b20 con 0x7f3f580a5520 2024-09-17T15:48:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f577fe640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f3f58135b50 legacy=0x7f3f581320d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6789/0 2024-09-17T15:48:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5e56b640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f580a0790 con 0x7f3f58135f70 2024-09-17T15:48:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5e56b640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5cb08640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f3f58135f70 0x7f3f581d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:56848/0 (socket says 172.21.15.64:56848) 2024-09-17T15:48:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5e56b640 10 monclient(hunting): authenticate will time out at 900.299994s 2024-09-17T15:48:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.624+0000 7f3f5cb08640 1 -- 172.21.15.64:0/4073341712 learned_addr learned my addr 172.21.15.64:0/4073341712 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2846769367 0 0) 0x7f3f580a0790 con 0x7f3f58135f70 2024-09-17T15:48:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient(hunting): my global_id is 5341 2024-09-17T15:48:22.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f44003520 con 0x7f3f58135f70 2024-09-17T15:48:22.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2340661630 0 0) 0x7f3f44003520 con 0x7f3f58135f70 2024-09-17T15:48:22.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f580a0790 con 0x7f3f58135f70 2024-09-17T15:48:22.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3f50004660 con 0x7f3f58135f70 2024-09-17T15:48:22.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:22.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:22.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient(hunting): dump: 2024-09-17T15:48:22.641 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.641 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.641 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.641 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.641 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4036577565 0 0) 0x7f3f58134b20 con 0x7f3f580a5520 2024-09-17T15:48:22.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient(hunting): my global_id is 5284 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f44003520 con 0x7f3f580a5520 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2575396056 0 0) 0x7f3f580a0790 con 0x7f3f58135f70 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 >> v1:172.21.15.100:6789/0 conn(0x7f3f580a5520 legacy=0x7f3f581319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 >> v1:172.21.15.64:6789/0 conn(0x7f3f58135b50 legacy=0x7f3f581320d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient: _finish_hunting 0 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 monclient: found mon.c 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f581327e0 con 0x7f3f58135f70 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient: _finish_auth 0 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:48:22.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:22.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f5e56b640 5 monclient: authenticate success, global_id 5341 2024-09-17T15:48:22.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f5e56b640 10 monclient: _renew_subs 2024-09-17T15:48:22.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f5e56b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:22.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f5e56b640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f5809fef0 con 0x7f3f58135f70 2024-09-17T15:48:22.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f5e56b640 10 monclient: _renew_subs 2024-09-17T15:48:22.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f5e56b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:48:22.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f5e56b640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f5809eb10 con 0x7f3f58135f70 2024-09-17T15:48:22.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f50004880 con 0x7f3f58135f70 2024-09-17T15:48:22.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:22.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3f50004e30 con 0x7f3f58135f70 2024-09-17T15:48:22.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:22.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f567fc640 4 set_mon_vals no callback set 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.625+0000 7f3f54ff9640 10 monclient: dump: 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.645 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.626+0000 7f3f5e56b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.626+0000 7f3f5e56b640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3f380065c0 con 0x7f3f38002860 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.626+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f3f50004fd0 con 0x7f3f58135f70 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.626+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(114..114 src has 1..114) ==== 3898+0+0 (unknown 642482365 0 0) 0x7f3f5004ef50 con 0x7f3f58135f70 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.626+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2395880718 0 0) 0x7f3f380065c0 con 0x7f3f38002860 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.626+0000 7f3f54ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.626+0000 7f3f54ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.626+0000 7f3f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.626+0000 7f3f54ff9640 10 monclient(hunting): my global_id is 5293 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f580a0790 con 0x7f3f38002860 2024-09-17T15:48:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f577fe640 10 monclient: get_auth_request con 0x7f3f440402b0 auth_method 0 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2307489687 0 0) 0x7f3f580a0790 con 0x7f3f38002860 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f380065c0 con 0x7f3f38002860 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3f48001720 con 0x7f3f38002860 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 10 monclient: dump: 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.647 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.648 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.648 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.648 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.648 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.648 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.648 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.648 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.648 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 794652181 0 0) 0x7f3f380065c0 con 0x7f3f38002860 2024-09-17T15:48:22.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.100:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f3800a160 con 0x7f3f38002860 2024-09-17T15:48:22.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3f48001940 con 0x7f3f38002860 2024-09-17T15:48:22.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:22.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 >> v1:172.21.15.100:6789/0 conn(0x7f3f38002860 legacy=0x7f3f38004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.627+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 >> v1:172.21.15.100:6789/0 conn(0x7f3f38002860 legacy=0x7f3f38004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:22.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.633+0000 7f3f5e56b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:22.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.633+0000 7f3f5e56b640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3f38009e50 con 0x7f3f38009110 2024-09-17T15:48:22.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2119190648 0 0) 0x7f3f38009e50 con 0x7f3f38009110 2024-09-17T15:48:22.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 10 monclient(hunting): my global_id is 5302 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f380065c0 con 0x7f3f38009110 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 934089470 0 0) 0x7f3f380065c0 con 0x7f3f38009110 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f38009e50 con 0x7f3f38009110 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3f48005340 con 0x7f3f38009110 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 10 monclient: dump: 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.651 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.652 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1684292251 0 0) 0x7f3f38009e50 con 0x7f3f38009110 2024-09-17T15:48:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.634+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 --> v1:172.21.15.100:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3f3800c3f0 con 0x7f3f38009110 2024-09-17T15:48:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7669 (unknown 0 0 2317783713) 0x7f3f48005340 con 0x7f3f38009110 2024-09-17T15:48:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f54ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f54ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 >> v1:172.21.15.100:6789/0 conn(0x7f3f38009110 legacy=0x7f3f380095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f54ff9640 1 -- 172.21.15.64:0/4073341712 >> v1:172.21.15.100:6789/0 conn(0x7f3f38009110 legacy=0x7f3f380095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:22.653 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:22.653 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:22.653 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:22.653 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "leader", 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 20, 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 159286, 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.663 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:22.663 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:22.663 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.663 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.663 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.663 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:22.663 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:22.663 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.663 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:22.665 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.665 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.665 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.665 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:22.665 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.665 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.665 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.665 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:22.665 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:22.665 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.665 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:48:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:48:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.673 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.673 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:22.673 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:22.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f5e56b640 1 -- 172.21.15.64:0/4073341712 >> v1:172.21.15.100:6812/4210963484 conn(0x7f3f440402b0 legacy=0x7f3f44042770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f5e56b640 10 monclient: shutdown 2024-09-17T15:48:22.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f5e56b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f5e56b640 1 -- 172.21.15.64:0/4073341712 >> v1:172.21.15.64:6791/0 conn(0x7f3f58135f70 legacy=0x7f3f581d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f5cb08640 1 -- 172.21.15.64:0/4073341712 reap_dead start 2024-09-17T15:48:22.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f5cb08640 1 -- 172.21.15.64:0/4073341712 reap_dead start 2024-09-17T15:48:22.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f5e56b640 1 -- 172.21.15.64:0/4073341712 shutdown_connections 2024-09-17T15:48:22.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f5e56b640 1 -- 172.21.15.64:0/4073341712 >> 172.21.15.64:0/4073341712 conn(0x7f3f5806db30 msgr2=0x7f3f5812c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f5e56b640 1 -- 172.21.15.64:0/4073341712 shutdown_connections 2024-09-17T15:48:22.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.635+0000 7f3f5e56b640 1 -- 172.21.15.64:0/4073341712 wait complete. 2024-09-17T15:48:22.675 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:48:22.675 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon g 2024-09-17T15:48:22.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-17T15:48:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.775+0000 7f5b10589640 10 monclient: get_monmap_and_config 2024-09-17T15:48:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.775+0000 7f5b10589640 10 monclient: build_initial_monmap 2024-09-17T15:48:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.775+0000 7f5b10589640 10 monclient: monmap: 2024-09-17T15:48:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:22.776457+0000 2024-09-17T15:48:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:22.776457+0000 2024-09-17T15:48:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:22.778 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.776+0000 7f5b10589640 1 Processor -- start 2024-09-17T15:48:22.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.776+0000 7f5b10589640 1 -- start start 2024-09-17T15:48:22.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.776+0000 7f5b10589640 10 monclient: init 2024-09-17T15:48:22.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.776+0000 7f5b10589640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:22.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.776+0000 7f5b10589640 10 monclient: _add_conns ranks=[1,5,8,6,4,2,0,7,3] 2024-09-17T15:48:22.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.776+0000 7f5b10589640 10 monclient(hunting): picked mon.noname-f con 0x7f5b080a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:22.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.776+0000 7f5b10589640 10 monclient(hunting): picked mon.noname-h con 0x7f5b08135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:22.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.776+0000 7f5b10589640 10 monclient(hunting): picked mon.noname-e con 0x7f5b08135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:22.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.776+0000 7f5b10589640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b0809d6f0 con 0x7f5b080a5520 2024-09-17T15:48:22.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.776+0000 7f5b10589640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b0809d9d0 con 0x7f5b08135b50 2024-09-17T15:48:22.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.776+0000 7f5b10589640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b0809dcb0 con 0x7f5b08135f70 2024-09-17T15:48:22.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.776+0000 7f5b10589640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:22.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.776+0000 7f5b10589640 10 monclient(hunting): authenticate will time out at 900.451940s 2024-09-17T15:48:22.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0eb26640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f5b08135f70 0x7f5b081415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:44394/0 (socket says 172.21.15.64:44394) 2024-09-17T15:48:22.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0db24640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7f5b08135b50 legacy=0x7f5b0813deb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:48:22.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0eb26640 1 -- 172.21.15.64:0/1098356617 learned_addr learned my addr 172.21.15.64:0/1098356617 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:22.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 1 -- 172.21.15.64:0/1098356617 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 132412459 0 0) 0x7f5b0809dcb0 con 0x7f5b08135f70 2024-09-17T15:48:22.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient(hunting): my global_id is 5273 2024-09-17T15:48:22.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 1 -- 172.21.15.64:0/1098356617 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5af4003520 con 0x7f5b08135f70 2024-09-17T15:48:22.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 1 -- 172.21.15.64:0/1098356617 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2998150425 0 0) 0x7f5af4003520 con 0x7f5b08135f70 2024-09-17T15:48:22.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 1 -- 172.21.15.64:0/1098356617 >> v1:172.21.15.100:6791/0 conn(0x7f5b08135b50 legacy=0x7f5b0813deb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 1 -- 172.21.15.64:0/1098356617 >> v1:172.21.15.100:6789/0 conn(0x7f5b080a5520 legacy=0x7f5b080a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient: _finish_hunting 0 2024-09-17T15:48:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 1 monclient: found mon.noname-e 2024-09-17T15:48:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:48:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 1 -- 172.21.15.64:0/1098356617 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b0809ccd0 con 0x7f5b08135f70 2024-09-17T15:48:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient: _finish_auth 0 2024-09-17T15:48:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient: _check_auth_tickets 2024-09-17T15:48:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 1 -- 172.21.15.64:0/1098356617 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5b00001bb0 con 0x7f5b08135f70 2024-09-17T15:48:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b10589640 5 monclient: authenticate success, global_id 5273 2024-09-17T15:48:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b10589640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient: dump: 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.789 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b10589640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 1 -- 172.21.15.64:0/1098356617 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5b00001dd0 con 0x7f5b08135f70 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 1 -- 172.21.15.64:0/1098356617 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5b000020c0 con 0x7f5b08135f70 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b10589640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b10589640 4 set_mon_vals no callback set 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.777+0000 7f5b0d323640 10 monclient: dump: 2024-09-17T15:48:22.790 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.791 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.778+0000 7f5b10589640 10 monclient: shutdown 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.778+0000 7f5b10589640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.778+0000 7f5b10589640 1 -- 172.21.15.64:0/1098356617 >> v1:172.21.15.64:6793/0 conn(0x7f5b08135f70 legacy=0x7f5b081415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.778+0000 7f5b10589640 1 -- 172.21.15.64:0/1098356617 shutdown_connections 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.778+0000 7f5b10589640 1 -- 172.21.15.64:0/1098356617 >> 172.21.15.64:0/1098356617 conn(0x7f5b0806db30 msgr2=0x7f5b080ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.778+0000 7f5b10589640 1 -- 172.21.15.64:0/1098356617 shutdown_connections 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.778+0000 7f5b10589640 1 -- 172.21.15.64:0/1098356617 wait complete. 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.778+0000 7f5b10589640 10 monclient: build_initial_monmap 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.778+0000 7f5b10589640 10 monclient: monmap: 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:22.792 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:22.793 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b10589640 1 Processor -- start 2024-09-17T15:48:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b10589640 1 -- start start 2024-09-17T15:48:22.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b10589640 10 monclient: init 2024-09-17T15:48:22.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b10589640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:22.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b10589640 10 monclient: _add_conns ranks=[6,3,0,5,7,1,8,2,4] 2024-09-17T15:48:22.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b10589640 10 monclient(hunting): picked mon.noname-d con 0x7f5b080a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:22.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b10589640 10 monclient(hunting): picked mon.noname-g con 0x7f5b08135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:22.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b10589640 10 monclient(hunting): picked mon.noname-a con 0x7f5b08135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:22.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b10589640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b08135230 con 0x7f5b08135f70 2024-09-17T15:48:22.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b10589640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b0809fdf0 con 0x7f5b08135b50 2024-09-17T15:48:22.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b10589640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b080a52e0 con 0x7f5b080a5520 2024-09-17T15:48:22.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b10589640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:22.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b0e325640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7f5b080a5520 legacy=0x7f5b080a4bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:48:22.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b10589640 10 monclient(hunting): authenticate will time out at 900.454566s 2024-09-17T15:48:22.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b0eb26640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f5b08135f70 legacy=0x7f5b081d62b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6789/0 2024-09-17T15:48:22.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b0db24640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f5b08135b50 0x7f5b081d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:46734/0 (socket says 172.21.15.64:46734) 2024-09-17T15:48:22.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5b0db24640 1 -- 172.21.15.64:0/1865368362 learned_addr learned my addr 172.21.15.64:0/1865368362 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:22.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2246756102 0 0) 0x7f5af8003a80 con 0x7f5b08135b50 2024-09-17T15:48:22.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5afeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5afeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5afeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5afeffd640 10 monclient(hunting): my global_id is 114348 2024-09-17T15:48:22.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.779+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5aec003550 con 0x7f5b08135b50 2024-09-17T15:48:22.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3875207855 0 0) 0x7f5aec003550 con 0x7f5b08135b50 2024-09-17T15:48:22.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5af8003a80 con 0x7f5b08135b50 2024-09-17T15:48:22.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5af8004570 con 0x7f5b08135b50 2024-09-17T15:48:22.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:22.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:48:22.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 10 monclient(hunting): dump: 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.796 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1391230916 0 0) 0x7f5af8003a80 con 0x7f5b08135b50 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 >> v1:172.21.15.64:6792/0 conn(0x7f5b080a5520 legacy=0x7f5b080a4bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 >> v1:172.21.15.64:6789/0 conn(0x7f5b08135f70 legacy=0x7f5b081d62b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 10 monclient: _finish_hunting 0 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 1 monclient: found mon.g 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b08134e70 con 0x7f5b08135b50 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 10 monclient: _finish_auth 0 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5b10589640 5 monclient: authenticate success, global_id 114348 2024-09-17T15:48:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5b10589640 10 monclient: _renew_subs 2024-09-17T15:48:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5b10589640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5b10589640 1 -- 172.21.15.64:0/1865368362 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5b0809f820 con 0x7f5b08135b50 2024-09-17T15:48:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5b10589640 10 monclient: _renew_subs 2024-09-17T15:48:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5b10589640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:48:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5b10589640 1 -- 172.21.15.64:0/1865368362 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5b0809ee90 con 0x7f5b08135b50 2024-09-17T15:48:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5af8004790 con 0x7f5b08135b50 2024-09-17T15:48:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5af8004dc0 con 0x7f5b08135b50 2024-09-17T15:48:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5b0cb22640 4 set_mon_vals no callback set 2024-09-17T15:48:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.780+0000 7f5afeffd640 10 monclient: dump: 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.800 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.781+0000 7f5b10589640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.781+0000 7f5b10589640 1 -- 172.21.15.64:0/1865368362 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5af00065c0 con 0x7f5af0002860 2024-09-17T15:48:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.781+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f5af8004fe0 con 0x7f5b08135b50 2024-09-17T15:48:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.781+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 8 ==== osd_map(114..114 src has 1..114) ==== 3898+0+0 (unknown 642482365 0 0) 0x7f5af804efa0 con 0x7f5b08135b50 2024-09-17T15:48:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.781+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3084635539 0 0) 0x7f5af00065c0 con 0x7f5af0002860 2024-09-17T15:48:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.781+0000 7f5afeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.781+0000 7f5afeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.781+0000 7f5afeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.781+0000 7f5afeffd640 10 monclient(hunting): my global_id is 114357 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.781+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5af8003a80 con 0x7f5af0002860 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.781+0000 7f5b0eb26640 10 monclient: get_auth_request con 0x7f5aec040190 auth_method 0 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.782+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3931436480 0 0) 0x7f5af8003a80 con 0x7f5af0002860 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.782+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5af00065c0 con 0x7f5af0002860 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.782+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5b04001950 con 0x7f5af0002860 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.782+0000 7f5afeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.782+0000 7f5afeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.782+0000 7f5afeffd640 10 monclient: dump: 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.782+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1738518186 0 0) 0x7f5af00065c0 con 0x7f5af0002860 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.782+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 --> v1:172.21.15.100:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5af000a160 con 0x7f5af0002860 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.782+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5b040057c0 con 0x7f5af0002860 2024-09-17T15:48:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.782+0000 7f5afeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.782+0000 7f5afeffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.782+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 >> v1:172.21.15.100:6790/0 conn(0x7f5af0002860 legacy=0x7f5af0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.782+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 >> v1:172.21.15.100:6790/0 conn(0x7f5af0002860 legacy=0x7f5af0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:22.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.788+0000 7f5b10589640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:22.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.788+0000 7f5b10589640 1 -- 172.21.15.64:0/1865368362 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5af0009e70 con 0x7f5af0009130 2024-09-17T15:48:22.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3157197268 0 0) 0x7f5af0009e70 con 0x7f5af0009130 2024-09-17T15:48:22.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 10 monclient(hunting): my global_id is 114366 2024-09-17T15:48:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5af00065c0 con 0x7f5af0009130 2024-09-17T15:48:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1531642524 0 0) 0x7f5af00065c0 con 0x7f5af0009130 2024-09-17T15:48:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5af0009e70 con 0x7f5af0009130 2024-09-17T15:48:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5b04003c20 con 0x7f5af0009130 2024-09-17T15:48:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:48:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 10 monclient: dump: 2024-09-17T15:48:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1569640568 0 0) 0x7f5af0009e70 con 0x7f5af0009130 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.789+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 --> v1:172.21.15.100:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5af000c410 con 0x7f5af0009130 2024-09-17T15:48:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7499 (unknown 0 0 832829565) 0x7f5b04005970 con 0x7f5af0009130 2024-09-17T15:48:22.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5afeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:22.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5afeffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:22.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 >> v1:172.21.15.100:6790/0 conn(0x7f5af0009130 legacy=0x7f5af00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5afeffd640 1 -- 172.21.15.64:0/1865368362 >> v1:172.21.15.100:6790/0 conn(0x7f5af0009130 legacy=0x7f5af00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:22.807 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 20, 2024-09-17T15:48:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 26013, 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:22.810 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.812 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:22.813 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.814 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.816 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.817 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.819 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:22.820 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:22.822 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.823 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:22.825 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:22.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5b10589640 1 -- 172.21.15.64:0/1865368362 >> v1:172.21.15.100:6812/4210963484 conn(0x7f5aec040190 legacy=0x7f5aec042650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5b10589640 10 monclient: shutdown 2024-09-17T15:48:22.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5b10589640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5b10589640 1 -- 172.21.15.64:0/1865368362 >> v1:172.21.15.100:6790/0 conn(0x7f5b08135b50 legacy=0x7f5b081d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5b0eb26640 1 -- 172.21.15.64:0/1865368362 reap_dead start 2024-09-17T15:48:22.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5b0eb26640 1 -- 172.21.15.64:0/1865368362 reap_dead start 2024-09-17T15:48:22.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5b10589640 1 -- 172.21.15.64:0/1865368362 shutdown_connections 2024-09-17T15:48:22.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5b10589640 1 -- 172.21.15.64:0/1865368362 >> 172.21.15.64:0/1865368362 conn(0x7f5b0806db30 msgr2=0x7f5b080a68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5b10589640 1 -- 172.21.15.64:0/1865368362 shutdown_connections 2024-09-17T15:48:22.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.790+0000 7f5b10589640 1 -- 172.21.15.64:0/1865368362 wait complete. 2024-09-17T15:48:22.828 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:48:22.828 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon i 2024-09-17T15:48:22.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-17T15:48:22.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.927+0000 7fe40dcdd640 10 monclient: get_monmap_and_config 2024-09-17T15:48:22.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.927+0000 7fe40dcdd640 10 monclient: build_initial_monmap 2024-09-17T15:48:22.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.927+0000 7fe40dcdd640 10 monclient: monmap: 2024-09-17T15:48:22.929 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:48:22.928671+0000 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:48:22.928671+0000 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:22.931 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.928+0000 7fe40dcdd640 1 Processor -- start 2024-09-17T15:48:22.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.928+0000 7fe40dcdd640 1 -- start start 2024-09-17T15:48:22.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.928+0000 7fe40dcdd640 10 monclient: init 2024-09-17T15:48:22.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe40dcdd640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:22.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe40dcdd640 10 monclient: _add_conns ranks=[1,5,0,3,7,6,4,2,8] 2024-09-17T15:48:22.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe40dcdd640 10 monclient(hunting): picked mon.noname-f con 0x7fe4080a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:48:22.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe40dcdd640 10 monclient(hunting): picked mon.noname-h con 0x7fe408135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:48:22.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe40dcdd640 10 monclient(hunting): picked mon.noname-a con 0x7fe408135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:48:22.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe40dcdd640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe40809d6f0 con 0x7fe408135f70 2024-09-17T15:48:22.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe40dcdd640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe40809d9d0 con 0x7fe4080a5520 2024-09-17T15:48:22.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe40dcdd640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe40809dcb0 con 0x7fe408135b50 2024-09-17T15:48:22.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe40dcdd640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:22.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe40dcdd640 10 monclient(hunting): authenticate will time out at 900.604223s 2024-09-17T15:48:22.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe407fff640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7fe408135f70 legacy=0x7fe4081415e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-09-17T15:48:22.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe406ffd640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7fe408135b50 legacy=0x7fe40813deb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:48:22.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe4077fe640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fe4080a5520 0x7fe4080a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.64:42706/0 (socket says 172.21.15.64:42706) 2024-09-17T15:48:22.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe4077fe640 1 -- 172.21.15.64:0/373297920 learned_addr learned my addr 172.21.15.64:0/373297920 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:22.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe4067fc640 1 -- 172.21.15.64:0/373297920 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3393431808 0 0) 0x7fe40809d9d0 con 0x7fe4080a5520 2024-09-17T15:48:22.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe4067fc640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe4067fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe4067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe4067fc640 10 monclient(hunting): my global_id is 5320 2024-09-17T15:48:22.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.929+0000 7fe4067fc640 1 -- 172.21.15.64:0/373297920 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3f0003520 con 0x7fe4080a5520 2024-09-17T15:48:22.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 1 -- 172.21.15.64:0/373297920 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4263759325 0 0) 0x7fe3f0003520 con 0x7fe4080a5520 2024-09-17T15:48:22.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 1 -- 172.21.15.64:0/373297920 >> v1:172.21.15.100:6791/0 conn(0x7fe408135b50 legacy=0x7fe40813deb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 1 -- 172.21.15.64:0/373297920 >> v1:172.21.15.64:6789/0 conn(0x7fe408135f70 legacy=0x7fe4081415e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 1 monclient: found mon.noname-f 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.100:6789/0 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 1 -- 172.21.15.64:0/373297920 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40809ccd0 con 0x7fe4080a5520 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 10 monclient: _finish_auth 0 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 10 monclient: _check_auth_tickets 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe40dcdd640 5 monclient: authenticate success, global_id 5320 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe40dcdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 1 -- 172.21.15.64:0/373297920 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe3f8001830 con 0x7fe4080a5520 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:48:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 10 monclient: dump: 2024-09-17T15:48:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.942 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe40dcdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 1 -- 172.21.15.64:0/373297920 <== mon.1 v1:172.21.15.100:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe3f8001a50 con 0x7fe4080a5520 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe40dcdd640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe40dcdd640 4 set_mon_vals no callback set 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 1 -- 172.21.15.64:0/373297920 <== mon.1 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe3f8001d40 con 0x7fe4080a5520 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe4067fc640 10 monclient: dump: 2024-09-17T15:48:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.944 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.945 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe40dcdd640 10 monclient: shutdown 2024-09-17T15:48:22.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe40dcdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe40dcdd640 1 -- 172.21.15.64:0/373297920 >> v1:172.21.15.100:6789/0 conn(0x7fe4080a5520 legacy=0x7fe4080a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe40dcdd640 1 -- 172.21.15.64:0/373297920 shutdown_connections 2024-09-17T15:48:22.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe40dcdd640 1 -- 172.21.15.64:0/373297920 >> 172.21.15.64:0/373297920 conn(0x7fe40806db30 msgr2=0x7fe4080ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe40dcdd640 1 -- 172.21.15.64:0/373297920 shutdown_connections 2024-09-17T15:48:22.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.930+0000 7fe40dcdd640 1 -- 172.21.15.64:0/373297920 wait complete. 2024-09-17T15:48:22.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.931+0000 7fe40dcdd640 10 monclient: build_initial_monmap 2024-09-17T15:48:22.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.931+0000 7fe40dcdd640 10 monclient: monmap: 2024-09-17T15:48:22.945 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:48:22.945 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:22.945 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:48:22.946 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:48:22.946 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:22.946 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:48:22.946 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:48:22.946 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:48:22.946 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:48:22.946 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:48:22.946 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:48:22.946 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:48:22.946 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:48:22.946 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:48:22.946 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.931+0000 7fe40dcdd640 1 Processor -- start 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.931+0000 7fe40dcdd640 1 -- start start 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.931+0000 7fe40dcdd640 10 monclient: init 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe40dcdd640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe40dcdd640 10 monclient: _add_conns ranks=[6,3,8,0,2,1,5,4,7] 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe40dcdd640 10 monclient(hunting): picked mon.noname-d con 0x7fe4080a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe40dcdd640 10 monclient(hunting): picked mon.noname-g con 0x7fe408135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe40dcdd640 10 monclient(hunting): picked mon.noname-e con 0x7fe408135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe40dcdd640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4080a1010 con 0x7fe408135b50 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe40dcdd640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe408135360 con 0x7fe4080a5520 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe40dcdd640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4080a3370 con 0x7fe408135f70 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe40dcdd640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:22.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe40dcdd640 10 monclient(hunting): authenticate will time out at 900.607285s 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe4077fe640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7fe4080a5520 legacy=0x7fe4080a48a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe407fff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fe408135f70 0x7fe4081d5e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:44406/0 (socket says 172.21.15.64:44406) 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe407fff640 1 -- 172.21.15.64:0/2873996975 learned_addr learned my addr 172.21.15.64:0/2873996975 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 989205705 0 0) 0x7fe4080a3370 con 0x7fe408135f70 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 10 monclient(hunting): my global_id is 5282 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3f4003520 con 0x7fe408135f70 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1806967692 0 0) 0x7fe3f4003520 con 0x7fe408135f70 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4080a3370 con 0x7fe408135f70 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe4000045d0 con 0x7fe408135f70 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:48:22.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 10 monclient(hunting): dump: 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.949 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.950 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.950 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.950 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:22.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2216470750 0 0) 0x7fe4080a1010 con 0x7fe408135b50 2024-09-17T15:48:22.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 10 monclient(hunting): my global_id is 114375 2024-09-17T15:48:22.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.932+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3f4003520 con 0x7fe408135b50 2024-09-17T15:48:22.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 690542097 0 0) 0x7fe4080a3370 con 0x7fe408135f70 2024-09-17T15:48:22.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 >> v1:172.21.15.64:6792/0 conn(0x7fe4080a5520 legacy=0x7fe4080a48a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 >> v1:172.21.15.100:6790/0 conn(0x7fe408135b50 legacy=0x7fe4081d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 10 monclient: _finish_hunting 0 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 1 monclient: found mon.e 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4080a2ad0 con 0x7fe408135f70 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 10 monclient: _finish_auth 0 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 10 monclient: _check_auth_tickets 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe40dcdd640 5 monclient: authenticate success, global_id 5282 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe40dcdd640 10 monclient: _renew_subs 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe40dcdd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe40dcdd640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe4080a1b70 con 0x7fe408135f70 2024-09-17T15:48:22.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe40dcdd640 10 monclient: _renew_subs 2024-09-17T15:48:22.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe40dcdd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:48:22.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe40dcdd640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fe4080a5190 con 0x7fe408135f70 2024-09-17T15:48:22.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe4000047f0 con 0x7fe408135f70 2024-09-17T15:48:22.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:22.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe400004da0 con 0x7fe408135f70 2024-09-17T15:48:22.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe405ffb640 4 set_mon_vals no callback set 2024-09-17T15:48:22.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:48:22.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe3effff640 10 monclient: dump: 2024-09-17T15:48:22.952 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.952 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.952 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.952 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe40dcdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.933+0000 7fe40dcdd640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe3e80065c0 con 0x7fe3e8002860 2024-09-17T15:48:22.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fe400004fc0 con 0x7fe408135f70 2024-09-17T15:48:22.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(114..114 src has 1..114) ==== 3898+0+0 (unknown 642482365 0 0) 0x7fe40004eee0 con 0x7fe408135f70 2024-09-17T15:48:22.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1606144587 0 0) 0x7fe3e80065c0 con 0x7fe3e8002860 2024-09-17T15:48:22.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 10 monclient(hunting): my global_id is 5137 2024-09-17T15:48:22.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4080a3370 con 0x7fe3e8002860 2024-09-17T15:48:22.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe406ffd640 10 monclient: get_auth_request con 0x7fe3f4040260 auth_method 0 2024-09-17T15:48:22.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1065476471 0 0) 0x7fe4080a3370 con 0x7fe3e8002860 2024-09-17T15:48:22.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3e80065c0 con 0x7fe3e8002860 2024-09-17T15:48:22.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe3f80014f0 con 0x7fe3e8002860 2024-09-17T15:48:22.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:48:22.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.934+0000 7fe3effff640 10 monclient: dump: 2024-09-17T15:48:22.955 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.955 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.955 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.955 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.955 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.955 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.955 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.955 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.955 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.955 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.956 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.956 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.956 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.956 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.956 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.956 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.935+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1848218491 0 0) 0x7fe3e80065c0 con 0x7fe3e8002860 2024-09-17T15:48:22.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.935+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.100:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3e800a160 con 0x7fe3e8002860 2024-09-17T15:48:22.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.935+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe3f8001580 con 0x7fe3e8002860 2024-09-17T15:48:22.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.935+0000 7fe3effff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.935+0000 7fe3effff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:22.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.935+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 >> v1:172.21.15.100:6792/0 conn(0x7fe3e8002860 legacy=0x7fe3e8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.935+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 >> v1:172.21.15.100:6792/0 conn(0x7fe3e8002860 legacy=0x7fe3e8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:22.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.941+0000 7fe40dcdd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:22.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.941+0000 7fe40dcdd640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe3e8009e50 con 0x7fe3e8009110 2024-09-17T15:48:22.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2457170100 0 0) 0x7fe3e8009e50 con 0x7fe3e8009110 2024-09-17T15:48:22.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 10 monclient(hunting): _negotiate 2024-09-17T15:48:22.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 10 monclient(hunting): my global_id is 5146 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3e80065c0 con 0x7fe3e8009110 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3342412393 0 0) 0x7fe3e80065c0 con 0x7fe3e8009110 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3e8009e50 con 0x7fe3e8009110 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe3f8001a30 con 0x7fe3e8009110 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 10 monclient: dump: 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:48:22.959 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:48:22.960 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:48:22.960 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:48:22.960 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:48:22.960 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:48:22.960 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:48:22.960 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:48:22.960 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:48:22.960 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:48:22.960 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:48:22.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2934541886 0 0) 0x7fe3e8009e50 con 0x7fe3e8009110 2024-09-17T15:48:22.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 --> v1:172.21.15.100:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe3e800c3f0 con 0x7fe3e8009110 2024-09-17T15:48:22.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7667 (unknown 0 0 3009762063) 0x7fe3f80073c0 con 0x7fe3e8009110 2024-09-17T15:48:22.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:22.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:22.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 >> v1:172.21.15.100:6792/0 conn(0x7fe3e8009110 legacy=0x7fe3e80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.942+0000 7fe3effff640 1 -- 172.21.15.64:0/2873996975 >> v1:172.21.15.100:6792/0 conn(0x7fe3e8009110 legacy=0x7fe3e80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:22.961 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:48:22.961 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:22.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:22.962 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:48:22.962 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 20, 2024-09-17T15:48:22.962 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 159578, 2024-09-17T15:48:22.962 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:48:22.962 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:48:22.962 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:48:22.962 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:48:22.962 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:48:22.962 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:48:22.962 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.962 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:48:22.962 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.963 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.964 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.965 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:48:22.965 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:48:22.965 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:48:22.965 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:48:22.965 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:48:22.965 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:48:22.965 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:22.965 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:48:22.965 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:48:22.965 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:22.965 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:48:22.966 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.967 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:48:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.969 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:48:22.970 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.971 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.972 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:48:22.972 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:48:22.972 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.972 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.972 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:48:22.972 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:48:22.972 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.972 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.973 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.973 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:48:22.973 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.973 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.973 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:48:22.973 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:48:22.973 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.973 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.973 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:48:22.974 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:48:22.974 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.974 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.974 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.974 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:48:22.974 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.974 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.974 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:48:22.974 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:48:22.975 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.975 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.975 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:48:22.975 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:48:22.975 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.975 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.975 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.975 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:48:22.976 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:48:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:48:22.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 4 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": [ 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:48:22.980 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:48:22.981 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:48:22.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.943+0000 7fe40dcdd640 1 -- 172.21.15.64:0/2873996975 >> v1:172.21.15.100:6812/4210963484 conn(0x7fe3f4040260 legacy=0x7fe3f4042720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.943+0000 7fe40dcdd640 10 monclient: shutdown 2024-09-17T15:48:22.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.943+0000 7fe40dcdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.943+0000 7fe40dcdd640 1 -- 172.21.15.64:0/2873996975 >> v1:172.21.15.64:6793/0 conn(0x7fe408135f70 legacy=0x7fe4081d5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.943+0000 7fe407fff640 1 -- 172.21.15.64:0/2873996975 reap_dead start 2024-09-17T15:48:22.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.943+0000 7fe407fff640 1 -- 172.21.15.64:0/2873996975 reap_dead start 2024-09-17T15:48:22.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.943+0000 7fe40dcdd640 1 -- 172.21.15.64:0/2873996975 shutdown_connections 2024-09-17T15:48:22.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.943+0000 7fe40dcdd640 1 -- 172.21.15.64:0/2873996975 >> 172.21.15.64:0/2873996975 conn(0x7fe40806db30 msgr2=0x7fe4080a6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.943+0000 7fe40dcdd640 1 -- 172.21.15.64:0/2873996975 shutdown_connections 2024-09-17T15:48:22.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:48:22.943+0000 7fe40dcdd640 1 -- 172.21.15.64:0/2873996975 wait complete. 2024-09-17T15:48:22.983 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:48:22.983 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-17T15:48:23.057 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36634]: starting. 2024-09-17T15:48:23.057 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36634]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36633 2024-09-17T15:48:23.057 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36634]: created object 0... 2024-09-17T15:48:23.057 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36634]: created object 25... 2024-09-17T15:48:23.057 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36634]: created object 49... 2024-09-17T15:48:23.057 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36634]: finishing. 2024-09-17T15:48:23.057 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36634]: shutting down. 2024-09-17T15:48:23.981 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36635]: starting. 2024-09-17T15:48:23.981 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36635]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36633 2024-09-17T15:48:23.981 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36635]: shutting down. 2024-09-17T15:48:26.069 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:26.070 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36670]: starting. 2024-09-17T15:48:26.070 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36670]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36633 2024-09-17T15:48:26.070 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36670]: created object 0... 2024-09-17T15:48:26.070 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36670]: created object 25... 2024-09-17T15:48:26.070 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36670]: created object 49... 2024-09-17T15:48:26.070 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36670]: finishing. 2024-09-17T15:48:26.070 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36670]: shutting down. 2024-09-17T15:48:26.081 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:26.081 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36672]: starting. 2024-09-17T15:48:26.081 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36672]: listing objects. 2024-09-17T15:48:26.081 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36672]: listed object 0... 2024-09-17T15:48:26.081 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36672]: listed object 25... 2024-09-17T15:48:26.081 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36672]: saw 50 objects 2024-09-17T15:48:26.081 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36672]: shutting down. 2024-09-17T15:48:27.002 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:27.002 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36671]: starting. 2024-09-17T15:48:27.002 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36671]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36633 2024-09-17T15:48:27.002 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36671]: shutting down. 2024-09-17T15:48:27.003 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:27.003 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:27.003 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:48:27.006 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:48:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36741]: starting. 2024-09-17T15:48:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36741]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36740 2024-09-17T15:48:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36741]: created object 0... 2024-09-17T15:48:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36741]: created object 25... 2024-09-17T15:48:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36741]: created object 49... 2024-09-17T15:48:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36741]: finishing. 2024-09-17T15:48:28.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36741]: shutting down. 2024-09-17T15:48:29.394 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36742]: starting. 2024-09-17T15:48:29.394 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36742]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36740 2024-09-17T15:48:29.394 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36742]: shutting down. 2024-09-17T15:48:31.494 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:31.494 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36777]: starting. 2024-09-17T15:48:31.494 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36777]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36740 2024-09-17T15:48:31.494 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36777]: created object 0... 2024-09-17T15:48:31.494 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36777]: created object 25... 2024-09-17T15:48:31.494 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36777]: created object 49... 2024-09-17T15:48:31.494 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36777]: finishing. 2024-09-17T15:48:31.494 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36777]: shutting down. 2024-09-17T15:48:32.436 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:32.436 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36779]: starting. 2024-09-17T15:48:32.436 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36779]: listing objects. 2024-09-17T15:48:32.436 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36779]: listed object 0... 2024-09-17T15:48:32.436 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36779]: listed object 25... 2024-09-17T15:48:32.436 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36779]: saw 50 objects 2024-09-17T15:48:32.436 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36779]: shutting down. 2024-09-17T15:48:33.419 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:33.419 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36778]: starting. 2024-09-17T15:48:33.419 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36778]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36740 2024-09-17T15:48:33.419 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36778]: shutting down. 2024-09-17T15:48:33.422 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:33.423 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:33.423 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:48:33.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:48:35.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36848]: starting. 2024-09-17T15:48:35.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36848]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36847 2024-09-17T15:48:35.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36848]: created object 0... 2024-09-17T15:48:35.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36848]: created object 25... 2024-09-17T15:48:35.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36848]: created object 49... 2024-09-17T15:48:35.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36848]: finishing. 2024-09-17T15:48:35.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36848]: shutting down. 2024-09-17T15:48:37.442 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36849]: starting. 2024-09-17T15:48:37.442 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36849]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36847 2024-09-17T15:48:37.442 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36849]: shutting down. 2024-09-17T15:48:39.498 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:39.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36884]: starting. 2024-09-17T15:48:39.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36884]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36847 2024-09-17T15:48:39.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36884]: created object 0... 2024-09-17T15:48:39.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36884]: created object 25... 2024-09-17T15:48:39.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36884]: created object 49... 2024-09-17T15:48:39.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36884]: finishing. 2024-09-17T15:48:39.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36884]: shutting down. 2024-09-17T15:48:39.509 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:39.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36886]: starting. 2024-09-17T15:48:39.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36886]: listing objects. 2024-09-17T15:48:39.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36886]: listed object 0... 2024-09-17T15:48:39.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36886]: listed object 25... 2024-09-17T15:48:39.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36886]: saw 50 objects 2024-09-17T15:48:39.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36886]: shutting down. 2024-09-17T15:48:40.397 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:40.397 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36885]: starting. 2024-09-17T15:48:40.397 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36885]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36847 2024-09-17T15:48:40.397 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36885]: shutting down. 2024-09-17T15:48:40.398 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:40.398 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:40.398 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:48:40.401 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:48:45.548 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36955]: starting. 2024-09-17T15:48:45.548 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36955]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36954 2024-09-17T15:48:45.549 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36955]: created object 0... 2024-09-17T15:48:45.549 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36955]: created object 25... 2024-09-17T15:48:45.549 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36955]: created object 49... 2024-09-17T15:48:45.549 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36955]: finishing. 2024-09-17T15:48:45.549 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[36955]: shutting down. 2024-09-17T15:48:46.479 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36956]: starting. 2024-09-17T15:48:46.479 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36956]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36954 2024-09-17T15:48:46.479 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[36956]: shutting down. 2024-09-17T15:48:51.496 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:51.496 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36991]: starting. 2024-09-17T15:48:51.496 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36991]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-36954 2024-09-17T15:48:51.496 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36991]: created object 0... 2024-09-17T15:48:51.497 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36991]: created object 25... 2024-09-17T15:48:51.497 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36991]: created object 49... 2024-09-17T15:48:51.497 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36991]: finishing. 2024-09-17T15:48:51.497 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[36991]: shutting down. 2024-09-17T15:48:51.507 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:51.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36993]: starting. 2024-09-17T15:48:51.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36993]: listing objects. 2024-09-17T15:48:51.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36993]: listed object 0... 2024-09-17T15:48:51.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36993]: listed object 25... 2024-09-17T15:48:51.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36993]: saw 50 objects 2024-09-17T15:48:51.507 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[36993]: shutting down. 2024-09-17T15:48:52.426 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:52.426 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36992]: starting. 2024-09-17T15:48:52.426 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36992]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-36954 2024-09-17T15:48:52.426 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[36992]: shutting down. 2024-09-17T15:48:52.428 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:52.428 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:52.428 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:48:52.430 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:48:54.479 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37062]: starting. 2024-09-17T15:48:54.480 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37062]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37061 2024-09-17T15:48:54.480 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37062]: created object 0... 2024-09-17T15:48:54.480 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37062]: created object 25... 2024-09-17T15:48:54.480 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37062]: created object 49... 2024-09-17T15:48:54.480 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37062]: finishing. 2024-09-17T15:48:54.480 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37062]: shutting down. 2024-09-17T15:48:55.402 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37063]: starting. 2024-09-17T15:48:55.402 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37063]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37061 2024-09-17T15:48:55.402 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37063]: shutting down. 2024-09-17T15:48:57.498 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:57.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37098]: starting. 2024-09-17T15:48:57.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37098]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37061 2024-09-17T15:48:57.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37098]: created object 0... 2024-09-17T15:48:57.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37098]: created object 25... 2024-09-17T15:48:57.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37098]: created object 49... 2024-09-17T15:48:57.499 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37098]: finishing. 2024-09-17T15:48:57.499 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37098]: shutting down. 2024-09-17T15:48:57.508 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:57.508 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37100]: starting. 2024-09-17T15:48:57.508 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37100]: listing objects. 2024-09-17T15:48:57.508 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37100]: listed object 0... 2024-09-17T15:48:57.508 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37100]: listed object 25... 2024-09-17T15:48:57.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37100]: saw 50 objects 2024-09-17T15:48:57.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37100]: shutting down. 2024-09-17T15:48:58.425 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:58.426 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37099]: starting. 2024-09-17T15:48:58.426 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37099]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37061 2024-09-17T15:48:58.426 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37099]: shutting down. 2024-09-17T15:48:58.428 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:58.428 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:48:58.428 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:48:58.431 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:49:00.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37169]: starting. 2024-09-17T15:49:00.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37169]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37168 2024-09-17T15:49:00.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37169]: created object 0... 2024-09-17T15:49:00.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37169]: created object 25... 2024-09-17T15:49:00.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37169]: created object 49... 2024-09-17T15:49:00.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37169]: finishing. 2024-09-17T15:49:00.520 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37169]: shutting down. 2024-09-17T15:49:01.439 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37170]: starting. 2024-09-17T15:49:01.439 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37170]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37168 2024-09-17T15:49:01.439 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37170]: shutting down. 2024-09-17T15:49:09.555 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:09.556 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37205]: starting. 2024-09-17T15:49:09.556 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37205]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37168 2024-09-17T15:49:09.556 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37205]: created object 0... 2024-09-17T15:49:09.556 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37205]: created object 25... 2024-09-17T15:49:09.556 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37205]: created object 49... 2024-09-17T15:49:09.556 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37205]: finishing. 2024-09-17T15:49:09.556 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37205]: shutting down. 2024-09-17T15:49:09.566 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:09.566 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37207]: starting. 2024-09-17T15:49:09.566 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37207]: listing objects. 2024-09-17T15:49:09.566 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37207]: listed object 0... 2024-09-17T15:49:09.566 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37207]: listed object 25... 2024-09-17T15:49:09.566 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37207]: saw 50 objects 2024-09-17T15:49:09.566 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37207]: shutting down. 2024-09-17T15:49:10.490 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:10.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37206]: starting. 2024-09-17T15:49:10.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37206]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37168 2024-09-17T15:49:10.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37206]: shutting down. 2024-09-17T15:49:10.491 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:10.491 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:10.491 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:49:10.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:49:12.587 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37276]: starting. 2024-09-17T15:49:12.587 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37276]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37275 2024-09-17T15:49:12.587 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37276]: created object 0... 2024-09-17T15:49:12.587 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37276]: created object 25... 2024-09-17T15:49:12.587 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37276]: created object 49... 2024-09-17T15:49:12.587 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37276]: finishing. 2024-09-17T15:49:12.588 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37276]: shutting down. 2024-09-17T15:49:14.513 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37277]: starting. 2024-09-17T15:49:14.513 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37277]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37275 2024-09-17T15:49:14.513 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37277]: shutting down. 2024-09-17T15:49:16.613 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:16.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37312]: starting. 2024-09-17T15:49:16.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37312]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37275 2024-09-17T15:49:16.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37312]: created object 0... 2024-09-17T15:49:16.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37312]: created object 25... 2024-09-17T15:49:16.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37312]: created object 49... 2024-09-17T15:49:16.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37312]: finishing. 2024-09-17T15:49:16.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37312]: shutting down. 2024-09-17T15:49:16.624 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:16.624 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37314]: starting. 2024-09-17T15:49:16.624 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37314]: listing objects. 2024-09-17T15:49:16.624 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37314]: listed object 0... 2024-09-17T15:49:16.624 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37314]: listed object 25... 2024-09-17T15:49:16.624 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37314]: saw 50 objects 2024-09-17T15:49:16.624 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37314]: shutting down. 2024-09-17T15:49:17.528 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:17.528 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37313]: starting. 2024-09-17T15:49:17.528 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37313]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37275 2024-09-17T15:49:17.528 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37313]: shutting down. 2024-09-17T15:49:17.529 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:17.529 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:17.529 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:49:17.532 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:49:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37383]: starting. 2024-09-17T15:49:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37383]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37382 2024-09-17T15:49:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37383]: created object 0... 2024-09-17T15:49:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37383]: created object 25... 2024-09-17T15:49:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37383]: created object 49... 2024-09-17T15:49:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37383]: finishing. 2024-09-17T15:49:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37383]: shutting down. 2024-09-17T15:49:20.550 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37384]: starting. 2024-09-17T15:49:20.551 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37384]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37382 2024-09-17T15:49:20.551 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37384]: shutting down. 2024-09-17T15:49:22.663 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:22.664 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37419]: starting. 2024-09-17T15:49:22.664 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37419]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37382 2024-09-17T15:49:22.664 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37419]: created object 0... 2024-09-17T15:49:22.664 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37419]: created object 25... 2024-09-17T15:49:22.664 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37419]: created object 49... 2024-09-17T15:49:22.664 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37419]: finishing. 2024-09-17T15:49:22.664 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37419]: shutting down. 2024-09-17T15:49:22.674 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:22.674 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37421]: starting. 2024-09-17T15:49:22.674 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37421]: listing objects. 2024-09-17T15:49:22.674 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37421]: listed object 0... 2024-09-17T15:49:22.674 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37421]: listed object 25... 2024-09-17T15:49:22.674 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37421]: saw 50 objects 2024-09-17T15:49:22.674 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37421]: shutting down. 2024-09-17T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37420]: starting. 2024-09-17T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37420]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37382 2024-09-17T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37420]: shutting down. 2024-09-17T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:49:23.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:49:27.745 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37490]: starting. 2024-09-17T15:49:27.745 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37490]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37489 2024-09-17T15:49:27.745 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37490]: created object 0... 2024-09-17T15:49:27.745 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37490]: created object 25... 2024-09-17T15:49:27.745 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37490]: created object 49... 2024-09-17T15:49:27.745 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37490]: finishing. 2024-09-17T15:49:27.745 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37490]: shutting down. 2024-09-17T15:49:28.662 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37491]: starting. 2024-09-17T15:49:28.662 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37491]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37489 2024-09-17T15:49:28.662 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37491]: shutting down. 2024-09-17T15:49:30.757 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:30.757 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37526]: starting. 2024-09-17T15:49:30.757 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37526]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37489 2024-09-17T15:49:30.757 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37526]: created object 0... 2024-09-17T15:49:30.757 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37526]: created object 25... 2024-09-17T15:49:30.757 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37526]: created object 49... 2024-09-17T15:49:30.758 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37526]: finishing. 2024-09-17T15:49:30.758 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37526]: shutting down. 2024-09-17T15:49:30.766 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:30.766 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37528]: starting. 2024-09-17T15:49:30.766 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37528]: listing objects. 2024-09-17T15:49:30.767 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37528]: listed object 0... 2024-09-17T15:49:30.767 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37528]: listed object 25... 2024-09-17T15:49:30.767 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37528]: saw 50 objects 2024-09-17T15:49:30.767 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37528]: shutting down. 2024-09-17T15:49:32.691 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:32.691 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37527]: starting. 2024-09-17T15:49:32.691 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37527]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37489 2024-09-17T15:49:32.691 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37527]: shutting down. 2024-09-17T15:49:32.692 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:32.692 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:32.692 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:49:32.695 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:49:34.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37597]: starting. 2024-09-17T15:49:34.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37597]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37596 2024-09-17T15:49:34.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37597]: created object 0... 2024-09-17T15:49:34.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37597]: created object 25... 2024-09-17T15:49:34.510 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37597]: created object 49... 2024-09-17T15:49:34.510 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37597]: finishing. 2024-09-17T15:49:34.510 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37597]: shutting down. 2024-09-17T15:49:35.423 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37598]: starting. 2024-09-17T15:49:35.423 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37598]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37596 2024-09-17T15:49:35.423 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37598]: shutting down. 2024-09-17T15:49:37.519 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:37.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37633]: starting. 2024-09-17T15:49:37.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37633]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37596 2024-09-17T15:49:37.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37633]: created object 0... 2024-09-17T15:49:37.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37633]: created object 25... 2024-09-17T15:49:37.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37633]: created object 49... 2024-09-17T15:49:37.520 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37633]: finishing. 2024-09-17T15:49:37.520 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37633]: shutting down. 2024-09-17T15:49:37.529 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37635]: starting. 2024-09-17T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37635]: listing objects. 2024-09-17T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37635]: listed object 0... 2024-09-17T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37635]: listed object 25... 2024-09-17T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37635]: saw 50 objects 2024-09-17T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37635]: shutting down. 2024-09-17T15:49:38.446 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:38.446 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37634]: starting. 2024-09-17T15:49:38.446 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37634]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37596 2024-09-17T15:49:38.446 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37634]: shutting down. 2024-09-17T15:49:38.447 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:38.447 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:38.447 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:49:38.450 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:49:40.552 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37704]: starting. 2024-09-17T15:49:40.552 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37704]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37703 2024-09-17T15:49:40.552 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37704]: created object 0... 2024-09-17T15:49:40.552 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37704]: created object 25... 2024-09-17T15:49:40.552 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37704]: created object 49... 2024-09-17T15:49:40.552 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37704]: finishing. 2024-09-17T15:49:40.552 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37704]: shutting down. 2024-09-17T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37705]: starting. 2024-09-17T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37705]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37703 2024-09-17T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37705]: shutting down. 2024-09-17T15:49:43.490 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:43.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37740]: starting. 2024-09-17T15:49:43.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37740]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37703 2024-09-17T15:49:43.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37740]: created object 0... 2024-09-17T15:49:43.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37740]: created object 25... 2024-09-17T15:49:43.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37740]: created object 49... 2024-09-17T15:49:43.490 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37740]: finishing. 2024-09-17T15:49:43.491 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37740]: shutting down. 2024-09-17T15:49:43.501 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:43.501 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37742]: starting. 2024-09-17T15:49:43.501 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37742]: listing objects. 2024-09-17T15:49:43.501 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37742]: listed object 0... 2024-09-17T15:49:43.501 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37742]: listed object 25... 2024-09-17T15:49:43.501 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37742]: saw 50 objects 2024-09-17T15:49:43.501 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37742]: shutting down. 2024-09-17T15:49:44.419 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:44.420 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37741]: starting. 2024-09-17T15:49:44.420 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37741]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37703 2024-09-17T15:49:44.420 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37741]: shutting down. 2024-09-17T15:49:44.421 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:44.421 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:44.421 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:49:44.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:49:46.564 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37811]: starting. 2024-09-17T15:49:46.565 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37811]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37810 2024-09-17T15:49:46.565 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37811]: created object 0... 2024-09-17T15:49:46.565 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37811]: created object 25... 2024-09-17T15:49:46.565 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37811]: created object 49... 2024-09-17T15:49:46.565 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37811]: finishing. 2024-09-17T15:49:46.565 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37811]: shutting down. 2024-09-17T15:49:47.484 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37812]: starting. 2024-09-17T15:49:47.484 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37812]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37810 2024-09-17T15:49:47.484 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37812]: shutting down. 2024-09-17T15:49:49.583 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:49.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37847]: starting. 2024-09-17T15:49:49.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37847]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37810 2024-09-17T15:49:49.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37847]: created object 0... 2024-09-17T15:49:49.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37847]: created object 25... 2024-09-17T15:49:49.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37847]: created object 49... 2024-09-17T15:49:49.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37847]: finishing. 2024-09-17T15:49:49.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[37847]: shutting down. 2024-09-17T15:49:49.593 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:49.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37849]: starting. 2024-09-17T15:49:49.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37849]: listing objects. 2024-09-17T15:49:49.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37849]: listed object 0... 2024-09-17T15:49:49.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37849]: listed object 25... 2024-09-17T15:49:49.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37849]: saw 50 objects 2024-09-17T15:49:49.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[37849]: shutting down. 2024-09-17T15:49:50.516 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:50.516 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37848]: starting. 2024-09-17T15:49:50.516 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37848]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37810 2024-09-17T15:49:50.516 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[37848]: shutting down. 2024-09-17T15:49:50.517 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:50.517 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:49:50.517 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:49:50.519 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:49:52.608 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37918]: starting. 2024-09-17T15:49:52.608 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37918]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37917 2024-09-17T15:49:52.608 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37918]: created object 0... 2024-09-17T15:49:52.609 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37918]: created object 25... 2024-09-17T15:49:52.609 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37918]: created object 49... 2024-09-17T15:49:52.609 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37918]: finishing. 2024-09-17T15:49:52.609 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[37918]: shutting down. 2024-09-17T15:49:52.984 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-17T15:49:52.984 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T15:49:52.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T15:49:52.987 INFO:tasks.ceph.mon.a:Started 2024-09-17T15:49:52.987 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-09-17T15:49:52.987 INFO:tasks.ceph.mon.d:Restarting daemon 2024-09-17T15:49:52.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-09-17T15:49:52.990 INFO:tasks.ceph.mon.d:Started 2024-09-17T15:49:52.990 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T15:49:52.990 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:49:52.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:49:52.992 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:49:52.993 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.h 2024-09-17T15:49:52.993 INFO:tasks.ceph.mon.h:Restarting daemon 2024-09-17T15:49:52.993 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-09-17T15:49:52.996 INFO:tasks.ceph.mon.h:Started 2024-09-17T15:49:52.996 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-17T15:49:52.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:49:53.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.122+0000 7f703ec03640 10 monclient: get_monmap_and_config 2024-09-17T15:49:53.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.122+0000 7f703ec03640 10 monclient: build_initial_monmap 2024-09-17T15:49:53.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.122+0000 7f703ec03640 10 monclient: monmap: 2024-09-17T15:49:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:49:53.123636+0000 2024-09-17T15:49:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:49:53.123636+0000 2024-09-17T15:49:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:53.125 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:53.126 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:53.126 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:53.126 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:53.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.123+0000 7f703ec03640 1 Processor -- start 2024-09-17T15:49:53.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.123+0000 7f703ec03640 1 -- start start 2024-09-17T15:49:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.123+0000 7f703ec03640 10 monclient: init 2024-09-17T15:49:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.124+0000 7f703ec03640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.124+0000 7f703ec03640 10 monclient: _add_conns ranks=[5,6,3,7,0,8,4,1,2] 2024-09-17T15:49:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.124+0000 7f703ec03640 10 monclient(hunting): picked mon.noname-h con 0x7f70380a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:49:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.124+0000 7f703ec03640 10 monclient(hunting): picked mon.noname-d con 0x7f7038135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:49:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.124+0000 7f7037fff640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f7038135c80 0x7f703813df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:41192/0 (socket says 172.21.15.64:41192) 2024-09-17T15:49:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.124+0000 7f703ec03640 10 monclient(hunting): picked mon.noname-g con 0x7f70381360a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:49:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.124+0000 7f703ec03640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70380aacb0 con 0x7f70381360a0 2024-09-17T15:49:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.124+0000 7f703ec03640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f703809d780 con 0x7f70380a54b0 2024-09-17T15:49:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.124+0000 7f703ec03640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f703809da60 con 0x7f7038135c80 2024-09-17T15:49:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.124+0000 7f703ec03640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.124+0000 7f703ec03640 10 monclient(hunting): authenticate will time out at 990.799629s 2024-09-17T15:49:53.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:53.124+0000 7f7037fff640 1 -- 172.21.15.64:0/578569025 learned_addr learned my addr 172.21.15.64:0/578569025 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:56.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 10 monclient(hunting): tick 2024-09-17T15:49:56.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:49:56.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 1 monclient(hunting): continuing hunt 2024-09-17T15:49:56.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:49:56.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 1 -- 172.21.15.64:0/578569025 >> v1:172.21.15.64:6792/0 conn(0x7f7038135c80 legacy=0x7f703813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:56.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 1 -- 172.21.15.64:0/578569025 >> v1:172.21.15.100:6791/0 conn(0x7f70380a54b0 legacy=0x7f70380a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:56.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 1 -- 172.21.15.64:0/578569025 >> v1:172.21.15.100:6790/0 conn(0x7f70381360a0 legacy=0x7f7038141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:56.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 10 monclient: _add_conns ranks=[1,7,2,0,4,8] 2024-09-17T15:49:56.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 10 monclient(hunting): picked mon.noname-f con 0x7f701c0037e0 addr v1:172.21.15.100:6789/0 2024-09-17T15:49:56.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 10 monclient(hunting): picked mon.noname-i con 0x7f701c007470 addr v1:172.21.15.100:6792/0 2024-09-17T15:49:56.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 10 monclient(hunting): picked mon.noname-b con 0x7f701c00b200 addr v1:172.21.15.64:6790/0 2024-09-17T15:49:56.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 1 -- 172.21.15.64:0/578569025 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f703809dd40 con 0x7f701c0037e0 2024-09-17T15:49:56.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 1 -- 172.21.15.64:0/578569025 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f701c010020 con 0x7f701c00b200 2024-09-17T15:49:56.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 1 -- 172.21.15.64:0/578569025 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f701c0112a0 con 0x7f701c007470 2024-09-17T15:49:56.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:56.124+0000 7f70367fc640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:58.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 1 -- 172.21.15.64:0/578569025 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1553720969 0 0) 0x7f701c0112a0 con 0x7f701c007470 2024-09-17T15:49:58.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient(hunting): my global_id is 5524 2024-09-17T15:49:58.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 1 -- 172.21.15.64:0/578569025 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7028003520 con 0x7f701c007470 2024-09-17T15:49:58.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 1 -- 172.21.15.64:0/578569025 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2380086996 0 0) 0x7f7028003520 con 0x7f701c007470 2024-09-17T15:49:58.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 1 -- 172.21.15.64:0/578569025 >> v1:172.21.15.64:6790/0 conn(0x7f701c00b200 legacy=0x7f701c00d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 1 -- 172.21.15.64:0/578569025 >> v1:172.21.15.100:6789/0 conn(0x7f701c0037e0 legacy=0x7f701c005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient: _finish_hunting 0 2024-09-17T15:49:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 1 monclient: found mon.noname-i 2024-09-17T15:49:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.100:6792/0 2024-09-17T15:49:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 1 -- 172.21.15.64:0/578569025 --> v1:172.21.15.100:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701c012580 con 0x7f701c007470 2024-09-17T15:49:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient: _finish_auth 0 2024-09-17T15:49:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient: _check_auth_tickets 2024-09-17T15:49:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 1 -- 172.21.15.64:0/578569025 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7024001870 con 0x7f701c007470 2024-09-17T15:49:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f703d1a0640 1 -- 172.21.15.64:0/578569025 reap_dead start 2024-09-17T15:49:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient: dump: 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f703ec03640 5 monclient: authenticate success, global_id 5524 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f703ec03640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 1 -- 172.21.15.64:0/578569025 <== mon.7 v1:172.21.15.100:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7024001a80 con 0x7f701c007470 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 1 -- 172.21.15.64:0/578569025 <== mon.7 v1:172.21.15.100:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7024001cf0 con 0x7f701c007470 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f703ec03640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f703ec03640 4 set_mon_vals no callback set 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.111+0000 7f70377fe640 10 monclient: dump: 2024-09-17T15:49:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.119 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.112+0000 7f703ec03640 10 monclient: shutdown 2024-09-17T15:49:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.112+0000 7f703ec03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.112+0000 7f703ec03640 1 -- 172.21.15.64:0/578569025 >> v1:172.21.15.100:6792/0 conn(0x7f701c007470 legacy=0x7f701c009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.112+0000 7f703ec03640 1 -- 172.21.15.64:0/578569025 shutdown_connections 2024-09-17T15:49:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.112+0000 7f703ec03640 1 -- 172.21.15.64:0/578569025 >> 172.21.15.64:0/578569025 conn(0x7f703806daa0 msgr2=0x7f70380ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.112+0000 7f703ec03640 1 -- 172.21.15.64:0/578569025 shutdown_connections 2024-09-17T15:49:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.112+0000 7f703ec03640 1 -- 172.21.15.64:0/578569025 wait complete. 2024-09-17T15:49:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.113+0000 7f703ec03640 10 monclient: build_initial_monmap 2024-09-17T15:49:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.113+0000 7f703ec03640 10 monclient: monmap: 2024-09-17T15:49:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:49:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:49:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:49:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:58.124 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.113+0000 7f703ec03640 1 Processor -- start 2024-09-17T15:49:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.113+0000 7f703ec03640 1 -- start start 2024-09-17T15:49:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.113+0000 7f703ec03640 10 monclient: init 2024-09-17T15:49:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.114+0000 7f703ec03640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.114+0000 7f703ec03640 10 monclient: _add_conns ranks=[4,6,1,7,5,2,0,8,3] 2024-09-17T15:49:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.114+0000 7f703ec03640 10 monclient(hunting): picked mon.noname-c con 0x7f701c007470 addr v1:172.21.15.64:6791/0 2024-09-17T15:49:58.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.114+0000 7f703ec03640 10 monclient(hunting): picked mon.noname-d con 0x7f703809f300 addr v1:172.21.15.64:6792/0 2024-09-17T15:49:58.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.114+0000 7f703ec03640 10 monclient(hunting): picked mon.noname-f con 0x7f703809f760 addr v1:172.21.15.100:6789/0 2024-09-17T15:49:58.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.114+0000 7f703ec03640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70380a2120 con 0x7f703809f760 2024-09-17T15:49:58.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.114+0000 7f703ec03640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70380a1b50 con 0x7f701c007470 2024-09-17T15:49:58.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.114+0000 7f703ec03640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70380a0660 con 0x7f703809f300 2024-09-17T15:49:58.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.114+0000 7f703ec03640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:58.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.114+0000 7f703ec03640 10 monclient(hunting): authenticate will time out at 995.789917s 2024-09-17T15:49:58.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.115+0000 7f7037fff640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f703809f300 0x7f70381d2f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:43958/0 (socket says 172.21.15.64:43958) 2024-09-17T15:49:58.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.115+0000 7f703c99f640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f701c007470 0x7f703809ebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:52692/0 (socket says 172.21.15.64:52692) 2024-09-17T15:49:58.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.115+0000 7f7037fff640 1 -- 172.21.15.64:0/3087364411 learned_addr learned my addr 172.21.15.64:0/3087364411 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:58.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 255881305 0 0) 0x7f70380a1b50 con 0x7f701c007470 2024-09-17T15:49:58.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient(hunting): my global_id is 5701 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f701c00ef70 con 0x7f701c007470 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1394661590 0 0) 0x7f701c010020 con 0x7f703809f760 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient(hunting): my global_id is 5761 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70380a1b50 con 0x7f703809f760 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 916246085 0 0) 0x7f701c00ef70 con 0x7f701c007470 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f701c010020 con 0x7f701c007470 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f702c001a00 con 0x7f701c007470 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:58.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient(hunting): dump: 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 487824341 0 0) 0x7f701c010020 con 0x7f701c007470 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 >> v1:172.21.15.64:6792/0 conn(0x7f703809f300 legacy=0x7f70381d2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 >> v1:172.21.15.100:6789/0 conn(0x7f703809f760 legacy=0x7f70381d66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient: _finish_hunting 0 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 1 monclient: found mon.c 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70380a29c0 con 0x7f701c007470 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient: _finish_auth 0 2024-09-17T15:49:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 10 monclient: _check_auth_tickets 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f70357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f703ec03640 5 monclient: authenticate success, global_id 5701 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f703ec03640 10 monclient: _renew_subs 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f703ec03640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f703ec03640 1 -- 172.21.15.64:0/3087364411 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70380a5230 con 0x7f701c007470 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f703ec03640 10 monclient: _renew_subs 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f703ec03640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.117+0000 7f703ec03640 1 -- 172.21.15.64:0/3087364411 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f70380a2ec0 con 0x7f701c007470 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.118+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f702c001d20 con 0x7f701c007470 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.118+0000 7f70357fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.118+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f702c003be0 con 0x7f701c007470 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.118+0000 7f70357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.118+0000 7f7036ffd640 4 set_mon_vals no callback set 2024-09-17T15:49:58.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.118+0000 7f70357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.118+0000 7f70357fa640 10 monclient: dump: 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.130 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.131 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.131 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.118+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f702c003d80 con 0x7f701c007470 2024-09-17T15:49:58.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.119+0000 7f703ec03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:58.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.119+0000 7f703ec03640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:58.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.119+0000 7f703ec03640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:49:58.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.119+0000 7f703ec03640 1 -- 172.21.15.64:0/3087364411 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7018005080 con 0x7f701c007470 2024-09-17T15:49:58.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.119+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(192..192 src has 1..192) ==== 3898+0+0 (unknown 454606166 0 0) 0x7f702c04eb10 con 0x7f701c007470 2024-09-17T15:49:58.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.119+0000 7f7037fff640 10 monclient: get_auth_request con 0x7f701c041a80 auth_method 0 2024-09-17T15:49:58.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.125+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f702c04ee20 con 0x7f701c007470 2024-09-17T15:49:58.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.125+0000 7f70357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:58.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.125+0000 7f70357fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:58.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.389+0000 7f703ec03640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:49:58.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.389+0000 7f703ec03640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:58.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.389+0000 7f703ec03640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:49:58.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.389+0000 7f703ec03640 1 -- 172.21.15.64:0/3087364411 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7018005370 con 0x7f701c007470 2024-09-17T15:49:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.390+0000 7f70357fa640 1 -- 172.21.15.64:0/3087364411 <== mon.4 v1:172.21.15.64:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 2647603102) 0x7f702c0176a0 con 0x7f701c007470 2024-09-17T15:49:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.390+0000 7f70357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:58.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.390+0000 7f70357fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:58.392 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":24,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:49:58.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.392+0000 7f703ec03640 1 -- 172.21.15.64:0/3087364411 >> v1:172.21.15.100:6812/4210963484 conn(0x7f701c041a80 legacy=0x7f701c043f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.392+0000 7f703ec03640 10 monclient: shutdown 2024-09-17T15:49:58.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.392+0000 7f703ec03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:58.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.392+0000 7f703ec03640 1 -- 172.21.15.64:0/3087364411 >> v1:172.21.15.64:6791/0 conn(0x7f701c007470 legacy=0x7f703809ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.392+0000 7f703ec03640 1 -- 172.21.15.64:0/3087364411 shutdown_connections 2024-09-17T15:49:58.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.392+0000 7f703ec03640 1 -- 172.21.15.64:0/3087364411 >> 172.21.15.64:0/3087364411 conn(0x7f703806daa0 msgr2=0x7f7038132630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:58.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.392+0000 7f703ec03640 1 -- 172.21.15.64:0/3087364411 shutdown_connections 2024-09-17T15:49:58.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.392+0000 7f703ec03640 1 -- 172.21.15.64:0/3087364411 wait complete. 2024-09-17T15:49:58.404 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-17T15:49:58.404 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-17T15:49:58.404 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:49:58.413 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37919]: starting. 2024-09-17T15:49:58.413 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37919]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37917 2024-09-17T15:49:58.413 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[37919]: shutting down. 2024-09-17T15:49:58.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.518+0000 7ff5cf63c640 10 monclient: get_monmap_and_config 2024-09-17T15:49:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.518+0000 7ff5cf63c640 10 monclient: build_initial_monmap 2024-09-17T15:49:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.518+0000 7ff5cf63c640 10 monclient: monmap: 2024-09-17T15:49:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:49:58.518962+0000 2024-09-17T15:49:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:49:58.518962+0000 2024-09-17T15:49:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:58.521 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:58.521 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:58.521 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:58.521 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:58.521 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.518+0000 7ff5cf63c640 1 Processor -- start 2024-09-17T15:49:58.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.518+0000 7ff5cf63c640 1 -- start start 2024-09-17T15:49:58.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.518+0000 7ff5cf63c640 10 monclient: init 2024-09-17T15:49:58.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5cf63c640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:58.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5cf63c640 10 monclient: _add_conns ranks=[8,1,5,6,7,2,0,4,3] 2024-09-17T15:49:58.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5cf63c640 10 monclient(hunting): picked mon.noname-e con 0x7ff5c80a54b0 addr v1:172.21.15.64:6793/0 2024-09-17T15:49:58.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5cf63c640 10 monclient(hunting): picked mon.noname-f con 0x7ff5c8135c80 addr v1:172.21.15.100:6789/0 2024-09-17T15:49:58.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5cf63c640 10 monclient(hunting): picked mon.noname-h con 0x7ff5c81360a0 addr v1:172.21.15.100:6791/0 2024-09-17T15:49:58.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5cf63c640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5c80aacb0 con 0x7ff5c8135c80 2024-09-17T15:49:58.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5cf63c640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5c809d780 con 0x7ff5c81360a0 2024-09-17T15:49:58.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5cf63c640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5c809da60 con 0x7ff5c80a54b0 2024-09-17T15:49:58.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5cf63c640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:58.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5cf63c640 10 monclient(hunting): authenticate will time out at 996.194415s 2024-09-17T15:49:58.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5cd3d8640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7ff5c80a54b0 0x7ff5c80a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:58012/0 (socket says 172.21.15.64:58012) 2024-09-17T15:49:58.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5cd3d8640 1 -- 172.21.15.64:0/4180896974 learned_addr learned my addr 172.21.15.64:0/4180896974 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:58.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 1 -- 172.21.15.64:0/4180896974 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1539352961 0 0) 0x7ff5c809da60 con 0x7ff5c80a54b0 2024-09-17T15:49:58.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 10 monclient(hunting): my global_id is 5624 2024-09-17T15:49:58.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 1 -- 172.21.15.64:0/4180896974 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5b0003520 con 0x7ff5c80a54b0 2024-09-17T15:49:58.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 1 -- 172.21.15.64:0/4180896974 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1556278248 0 0) 0x7ff5b0003520 con 0x7ff5c80a54b0 2024-09-17T15:49:58.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 1 -- 172.21.15.64:0/4180896974 >> v1:172.21.15.100:6791/0 conn(0x7ff5c81360a0 legacy=0x7ff5c8141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 1 -- 172.21.15.64:0/4180896974 >> v1:172.21.15.100:6789/0 conn(0x7ff5c8135c80 legacy=0x7ff5c813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:49:58.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 1 monclient: found mon.noname-e 2024-09-17T15:49:58.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:58.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:49:58.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 1 -- 172.21.15.64:0/4180896974 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5c809dd40 con 0x7ff5c80a54b0 2024-09-17T15:49:58.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 10 monclient: _finish_auth 0 2024-09-17T15:49:58.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:49:58.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.519+0000 7ff5c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:58.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5c7fff640 1 -- 172.21.15.64:0/4180896974 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff5c0001bb0 con 0x7ff5c80a54b0 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5cf63c640 5 monclient: authenticate success, global_id 5624 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5cf63c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5c7fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5c7fff640 10 monclient: dump: 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.531 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.532 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.532 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.532 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.532 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.532 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.532 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.532 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5cf63c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:58.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5c7fff640 1 -- 172.21.15.64:0/4180896974 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff5c0001dd0 con 0x7ff5c80a54b0 2024-09-17T15:49:58.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5c7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:58.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5c7fff640 1 -- 172.21.15.64:0/4180896974 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff5c0004f60 con 0x7ff5c80a54b0 2024-09-17T15:49:58.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5cf63c640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:58.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5cf63c640 4 set_mon_vals no callback set 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5c7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5c7fff640 10 monclient: dump: 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.533 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.534 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.534 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.534 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.534 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.534 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5cf63c640 10 monclient: shutdown 2024-09-17T15:49:58.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5cf63c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:58.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5cf63c640 1 -- 172.21.15.64:0/4180896974 >> v1:172.21.15.64:6793/0 conn(0x7ff5c80a54b0 legacy=0x7ff5c80a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5cf63c640 1 -- 172.21.15.64:0/4180896974 shutdown_connections 2024-09-17T15:49:58.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5cf63c640 1 -- 172.21.15.64:0/4180896974 >> 172.21.15.64:0/4180896974 conn(0x7ff5c806daa0 msgr2=0x7ff5c80ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:58.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5cf63c640 1 -- 172.21.15.64:0/4180896974 shutdown_connections 2024-09-17T15:49:58.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.520+0000 7ff5cf63c640 1 -- 172.21.15.64:0/4180896974 wait complete. 2024-09-17T15:49:58.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.521+0000 7ff5cf63c640 10 monclient: build_initial_monmap 2024-09-17T15:49:58.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.521+0000 7ff5cf63c640 10 monclient: monmap: 2024-09-17T15:49:58.535 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:58.535 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:58.535 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:49:58.535 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:49:58.535 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:58.535 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:49:58.535 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:58.535 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:58.535 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:58.535 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:58.535 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:58.535 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:58.536 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:58.536 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:58.536 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:58.536 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.521+0000 7ff5cf63c640 1 Processor -- start 2024-09-17T15:49:58.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.521+0000 7ff5cf63c640 1 -- start start 2024-09-17T15:49:58.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.521+0000 7ff5cf63c640 10 monclient: init 2024-09-17T15:49:58.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:58.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 10 monclient: _add_conns ranks=[3,5,6,1,4,7,2,0,8] 2024-09-17T15:49:58.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 10 monclient(hunting): picked mon.noname-g con 0x7ff5c80a54b0 addr v1:172.21.15.100:6790/0 2024-09-17T15:49:58.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 10 monclient(hunting): picked mon.noname-h con 0x7ff5c8135c80 addr v1:172.21.15.100:6791/0 2024-09-17T15:49:58.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 10 monclient(hunting): picked mon.noname-d con 0x7ff5c81360a0 addr v1:172.21.15.64:6792/0 2024-09-17T15:49:58.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5c80a7790 con 0x7ff5c80a54b0 2024-09-17T15:49:58.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5c812db80 con 0x7ff5c8135c80 2024-09-17T15:49:58.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5c8125be0 con 0x7ff5c81360a0 2024-09-17T15:49:58.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:58.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 10 monclient(hunting): authenticate will time out at 996.197190s 2024-09-17T15:49:58.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cdbd9640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7ff5c81360a0 0x7ff5c81d6a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:43966/0 (socket says 172.21.15.64:43966) 2024-09-17T15:49:58.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5ccbd7640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7ff5c8135c80 0x7ff5c81d3320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:34372/0 (socket says 172.21.15.64:34372) 2024-09-17T15:49:58.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cdbd9640 1 -- 172.21.15.64:0/929865854 learned_addr learned my addr 172.21.15.64:0/929865854 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:58.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1399913149 0 0) 0x7ff5c8125be0 con 0x7ff5c81360a0 2024-09-17T15:49:58.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): my global_id is 124107 2024-09-17T15:49:58.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5b4003520 con 0x7ff5c81360a0 2024-09-17T15:49:58.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 578609105 0 0) 0x7ff5c809dd40 con 0x7ff5c80a54b0 2024-09-17T15:49:58.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): my global_id is 114771 2024-09-17T15:49:58.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5c8125be0 con 0x7ff5c80a54b0 2024-09-17T15:49:58.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3401849529 0 0) 0x7ff5c812db80 con 0x7ff5c8135c80 2024-09-17T15:49:58.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): my global_id is 124133 2024-09-17T15:49:58.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5c809dd40 con 0x7ff5c8135c80 2024-09-17T15:49:58.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 657558435 0 0) 0x7ff5b4003520 con 0x7ff5c81360a0 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5c812db80 con 0x7ff5c81360a0 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff5b8001a50 con 0x7ff5c81360a0 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): dump: 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.539 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.540 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.540 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.540 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.540 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.540 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.540 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.540 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:49:58.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1040594869 0 0) 0x7ff5c8125be0 con 0x7ff5c80a54b0 2024-09-17T15:49:58.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5b4003520 con 0x7ff5c80a54b0 2024-09-17T15:49:58.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff5c0004ea0 con 0x7ff5c80a54b0 2024-09-17T15:49:58.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:58.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:49:58.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): dump: 2024-09-17T15:49:58.541 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.541 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.541 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.541 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.541 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.541 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.541 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.541 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.541 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.541 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.541 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.541 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2029832655 0 0) 0x7ff5c809dd40 con 0x7ff5c8135c80 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5c8125be0 con 0x7ff5c8135c80 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff5bc001990 con 0x7ff5c8135c80 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient(hunting): dump: 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.542 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.543 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.543 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.543 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.543 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.543 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.543 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.543 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.543 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.543 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.544 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3412586229 0 0) 0x7ff5c812db80 con 0x7ff5c81360a0 2024-09-17T15:49:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 >> v1:172.21.15.100:6791/0 conn(0x7ff5c8135c80 legacy=0x7ff5c81d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 >> v1:172.21.15.100:6790/0 conn(0x7ff5c80a54b0 legacy=0x7ff5c8141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:49:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 monclient: found mon.d 2024-09-17T15:49:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:49:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5c812de60 con 0x7ff5c81360a0 2024-09-17T15:49:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:49:58.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:49:58.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:58.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 5 monclient: authenticate success, global_id 124107 2024-09-17T15:49:58.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 10 monclient: _renew_subs 2024-09-17T15:49:58.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:49:58.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.522+0000 7ff5cf63c640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff5c8126b40 con 0x7ff5c81360a0 2024-09-17T15:49:58.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.523+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff5b80038c0 con 0x7ff5c81360a0 2024-09-17T15:49:58.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.523+0000 7ff5c5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:58.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.523+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff5b8003eb0 con 0x7ff5c81360a0 2024-09-17T15:49:58.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.523+0000 7ff5c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.523+0000 7ff5c5ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:49:58.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.523+0000 7ff5c5ffb640 10 monclient: dump: 2024-09-17T15:49:58.545 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.546 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.546 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.546 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.546 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.546 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.546 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.546 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.546 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.546 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.546 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.546 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.546 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.546 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.547 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.547 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.523+0000 7ff5cf63c640 10 monclient: _renew_subs 2024-09-17T15:49:58.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.523+0000 7ff5c77fe640 4 set_mon_vals no callback set 2024-09-17T15:49:58.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.523+0000 7ff5cf63c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:49:58.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.523+0000 7ff5cf63c640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7ff5c8124720 con 0x7ff5c81360a0 2024-09-17T15:49:58.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.523+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7ff5b80040d0 con 0x7ff5c81360a0 2024-09-17T15:49:58.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5cf63c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:58.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5cf63c640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff5c80647b0 con 0x7ff5c81424a0 2024-09-17T15:49:58.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(193..193 src has 1..193) ==== 3506+0+0 (unknown 1845487835 0 0) 0x7ff5b804eaa0 con 0x7ff5c81360a0 2024-09-17T15:49:58.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5cd3d8640 10 monclient: get_auth_request con 0x7ff5b4040c90 auth_method 0 2024-09-17T15:49:58.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 249643800 0 0) 0x7ff5c8125be0 con 0x7ff5c81424a0 2024-09-17T15:49:58.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 10 monclient(hunting): my global_id is 124101 2024-09-17T15:49:58.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5c812db80 con 0x7ff5c81424a0 2024-09-17T15:49:58.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 429647231 0 0) 0x7ff5c812db80 con 0x7ff5c81424a0 2024-09-17T15:49:58.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5c8125be0 con 0x7ff5c81424a0 2024-09-17T15:49:58.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff5bc003930 con 0x7ff5c81424a0 2024-09-17T15:49:58.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:58.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 10 monclient: dump: 2024-09-17T15:49:58.548 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.548 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.548 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.549 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.549 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.549 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.549 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.549 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.549 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.549 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.549 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.549 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.549 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.549 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.549 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.549 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 387624858 0 0) 0x7ff5c8125be0 con 0x7ff5c81424a0 2024-09-17T15:49:58.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.524+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.64:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5c80abea0 con 0x7ff5c81424a0 2024-09-17T15:49:58.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.525+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff5bc0057c0 con 0x7ff5c81424a0 2024-09-17T15:49:58.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.525+0000 7ff5c5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:58.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.525+0000 7ff5c5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:58.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.525+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 >> v1:172.21.15.64:6789/0 conn(0x7ff5c81424a0 legacy=0x7ff5c812a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.525+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 >> v1:172.21.15.64:6789/0 conn(0x7ff5c81424a0 legacy=0x7ff5c812a610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:58.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5cf63c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:58.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5cf63c640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff5c80a6a50 con 0x7ff5c802e450 2024-09-17T15:49:58.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1773319041 0 0) 0x7ff5c80a6a50 con 0x7ff5c802e450 2024-09-17T15:49:58.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 10 monclient(hunting): my global_id is 124110 2024-09-17T15:49:58.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5c8125be0 con 0x7ff5c802e450 2024-09-17T15:49:58.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3322078606 0 0) 0x7ff5c8125be0 con 0x7ff5c802e450 2024-09-17T15:49:58.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5c80a6a50 con 0x7ff5c802e450 2024-09-17T15:49:58.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff5bc004250 con 0x7ff5c802e450 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 10 monclient: dump: 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.553 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.554 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.554 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.554 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.554 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.554 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3295577558 0 0) 0x7ff5c80a6a50 con 0x7ff5c802e450 2024-09-17T15:49:58.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.531+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 --> v1:172.21.15.64:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff5c8068b30 con 0x7ff5c802e450 2024-09-17T15:49:58.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 3476037615) 0x7ff5bc004320 con 0x7ff5c802e450 2024-09-17T15:49:58.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5c5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:58.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5c5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:58.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 >> v1:172.21.15.64:6789/0 conn(0x7ff5c802e450 legacy=0x7ff5c812ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5c5ffb640 1 -- 172.21.15.64:0/929865854 >> v1:172.21.15.64:6789/0 conn(0x7ff5c802e450 legacy=0x7ff5c812ad20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:58.555 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:49:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:49:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:49:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:49:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:49:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 5449, 2024-09-17T15:49:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:49:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:49:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:49:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:49:58.555 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:58.556 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:49:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:58.557 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:49:58.558 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:49:58.559 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:49:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:49:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:49:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:49:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.561 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:49:58.562 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.563 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:49:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:49:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:49:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:49:58.565 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.566 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.568 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:49:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:49:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:49:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:49:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:49:58.569 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:49:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:49:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:49:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:49:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:49:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:49:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.572 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:49:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:49:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:49:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.573 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.574 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:49:58.574 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:49:58.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5cf63c640 1 -- 172.21.15.64:0/929865854 >> v1:172.21.15.100:6812/4210963484 conn(0x7ff5b4040c90 legacy=0x7ff5b4043150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5cf63c640 10 monclient: shutdown 2024-09-17T15:49:58.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5cf63c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:58.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5cf63c640 1 -- 172.21.15.64:0/929865854 >> v1:172.21.15.64:6792/0 conn(0x7ff5c81360a0 legacy=0x7ff5c81d6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5cdbd9640 1 -- 172.21.15.64:0/929865854 reap_dead start 2024-09-17T15:49:58.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5cdbd9640 1 -- 172.21.15.64:0/929865854 reap_dead start 2024-09-17T15:49:58.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5cf63c640 1 -- 172.21.15.64:0/929865854 shutdown_connections 2024-09-17T15:49:58.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5cf63c640 1 -- 172.21.15.64:0/929865854 >> 172.21.15.64:0/929865854 conn(0x7ff5c806daa0 msgr2=0x7ff5c80a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:58.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5cf63c640 1 -- 172.21.15.64:0/929865854 shutdown_connections 2024-09-17T15:49:58.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.532+0000 7ff5cf63c640 1 -- 172.21.15.64:0/929865854 wait complete. 2024-09-17T15:49:58.576 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:49:58.577 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-17T15:49:58.577 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:49:58.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.677+0000 7fbc9042a640 10 monclient: get_monmap_and_config 2024-09-17T15:49:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.677+0000 7fbc9042a640 10 monclient: build_initial_monmap 2024-09-17T15:49:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.677+0000 7fbc9042a640 10 monclient: monmap: 2024-09-17T15:49:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:49:58.677949+0000 2024-09-17T15:49:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:49:58.677949+0000 2024-09-17T15:49:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:58.680 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.677+0000 7fbc9042a640 1 Processor -- start 2024-09-17T15:49:58.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.677+0000 7fbc9042a640 1 -- start start 2024-09-17T15:49:58.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.677+0000 7fbc9042a640 10 monclient: init 2024-09-17T15:49:58.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc9042a640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:58.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc9042a640 10 monclient: _add_conns ranks=[7,1,4,0,3,6,8,2,5] 2024-09-17T15:49:58.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc9042a640 10 monclient(hunting): picked mon.noname-i con 0x7fbc880a54b0 addr v1:172.21.15.100:6792/0 2024-09-17T15:49:58.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc9042a640 10 monclient(hunting): picked mon.noname-f con 0x7fbc88135c80 addr v1:172.21.15.100:6789/0 2024-09-17T15:49:58.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc9042a640 10 monclient(hunting): picked mon.noname-c con 0x7fbc881360a0 addr v1:172.21.15.64:6791/0 2024-09-17T15:49:58.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc9042a640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc880aacb0 con 0x7fbc88135c80 2024-09-17T15:49:58.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc9042a640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc8809d780 con 0x7fbc881360a0 2024-09-17T15:49:58.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc9042a640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc8809da60 con 0x7fbc880a54b0 2024-09-17T15:49:58.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc9042a640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:58.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc9042a640 10 monclient(hunting): authenticate will time out at 996.353456s 2024-09-17T15:49:58.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc8e9c7640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fbc881360a0 0x7fbc88141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:52700/0 (socket says 172.21.15.64:52700) 2024-09-17T15:49:58.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc8e9c7640 1 -- 172.21.15.64:0/1343045097 learned_addr learned my addr 172.21.15.64:0/1343045097 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:58.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc8d1c4640 1 -- 172.21.15.64:0/1343045097 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2933325406 0 0) 0x7fbc8809d780 con 0x7fbc881360a0 2024-09-17T15:49:58.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc8d1c4640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc8d1c4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc8d1c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc8d1c4640 10 monclient(hunting): my global_id is 5728 2024-09-17T15:49:58.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.678+0000 7fbc8d1c4640 1 -- 172.21.15.64:0/1343045097 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc70003520 con 0x7fbc881360a0 2024-09-17T15:49:58.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 1 -- 172.21.15.64:0/1343045097 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 307967697 0 0) 0x7fbc880aacb0 con 0x7fbc88135c80 2024-09-17T15:49:58.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient(hunting): my global_id is 5815 2024-09-17T15:49:58.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 1 -- 172.21.15.64:0/1343045097 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc8809d780 con 0x7fbc88135c80 2024-09-17T15:49:58.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 1 -- 172.21.15.64:0/1343045097 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1042293146 0 0) 0x7fbc8809da60 con 0x7fbc880a54b0 2024-09-17T15:49:58.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient(hunting): my global_id is 5551 2024-09-17T15:49:58.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 1 -- 172.21.15.64:0/1343045097 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc880aacb0 con 0x7fbc880a54b0 2024-09-17T15:49:58.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 1 -- 172.21.15.64:0/1343045097 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1422718819 0 0) 0x7fbc70003520 con 0x7fbc881360a0 2024-09-17T15:49:58.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 1 -- 172.21.15.64:0/1343045097 >> v1:172.21.15.100:6792/0 conn(0x7fbc880a54b0 legacy=0x7fbc880a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 1 -- 172.21.15.64:0/1343045097 >> v1:172.21.15.100:6789/0 conn(0x7fbc88135c80 legacy=0x7fbc8813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient: _finish_hunting 0 2024-09-17T15:49:58.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 1 monclient: found mon.noname-c 2024-09-17T15:49:58.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:58.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:49:58.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 1 -- 172.21.15.64:0/1343045097 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc8809dd40 con 0x7fbc881360a0 2024-09-17T15:49:58.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient: _finish_auth 0 2024-09-17T15:49:58.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient: _check_auth_tickets 2024-09-17T15:49:58.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:58.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 1 -- 172.21.15.64:0/1343045097 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbc78001b30 con 0x7fbc881360a0 2024-09-17T15:49:58.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:49:58.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient: dump: 2024-09-17T15:49:58.693 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.693 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.694 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.694 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.694 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.694 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.694 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.694 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.694 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.694 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.694 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.694 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.694 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.694 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.695 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.695 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc9042a640 5 monclient: authenticate success, global_id 5728 2024-09-17T15:49:58.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc9042a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:58.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 1 -- 172.21.15.64:0/1343045097 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc78001cd0 con 0x7fbc881360a0 2024-09-17T15:49:58.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:58.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 1 -- 172.21.15.64:0/1343045097 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbc78004f60 con 0x7fbc881360a0 2024-09-17T15:49:58.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc9042a640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:58.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc9042a640 4 set_mon_vals no callback set 2024-09-17T15:49:58.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:58.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc8d1c4640 10 monclient: dump: 2024-09-17T15:49:58.696 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.696 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.696 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.696 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.696 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.696 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.696 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.696 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.696 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.696 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.696 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.697 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.697 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.697 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.697 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.697 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc9042a640 10 monclient: shutdown 2024-09-17T15:49:58.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc9042a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:58.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc9042a640 1 -- 172.21.15.64:0/1343045097 >> v1:172.21.15.64:6791/0 conn(0x7fbc881360a0 legacy=0x7fbc88141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc9042a640 1 -- 172.21.15.64:0/1343045097 shutdown_connections 2024-09-17T15:49:58.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc9042a640 1 -- 172.21.15.64:0/1343045097 >> 172.21.15.64:0/1343045097 conn(0x7fbc8806daa0 msgr2=0x7fbc880ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:58.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc9042a640 1 -- 172.21.15.64:0/1343045097 shutdown_connections 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.679+0000 7fbc9042a640 1 -- 172.21.15.64:0/1343045097 wait complete. 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.680+0000 7fbc9042a640 10 monclient: build_initial_monmap 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.680+0000 7fbc9042a640 10 monclient: monmap: 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.680+0000 7fbc9042a640 1 Processor -- start 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.680+0000 7fbc9042a640 1 -- start start 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.680+0000 7fbc9042a640 10 monclient: init 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc9042a640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc9042a640 10 monclient: _add_conns ranks=[2,6,0,8,4,3,5,7,1] 2024-09-17T15:49:58.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc9042a640 10 monclient(hunting): picked mon.noname-b con 0x7fbc880a54b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:49:58.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc9042a640 10 monclient(hunting): picked mon.noname-d con 0x7fbc88135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:49:58.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc9042a640 10 monclient(hunting): picked mon.noname-a con 0x7fbc881360a0 addr v1:172.21.15.64:6789/0 2024-09-17T15:49:58.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc9042a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc88142630 con 0x7fbc881360a0 2024-09-17T15:49:58.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc9042a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc880a0430 con 0x7fbc880a54b0 2024-09-17T15:49:58.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc9042a640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc880a0710 con 0x7fbc88135c80 2024-09-17T15:49:58.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc9042a640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:58.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc9042a640 10 monclient(hunting): authenticate will time out at 996.356269s 2024-09-17T15:49:58.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc8e1c6640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fbc880a54b0 0x7fbc88141810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46884/0 (socket says 172.21.15.64:46884) 2024-09-17T15:49:58.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc8e1c6640 1 -- 172.21.15.64:0/1471222660 learned_addr learned my addr 172.21.15.64:0/1471222660 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:58.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1683787218 0 0) 0x7fbc88142630 con 0x7fbc881360a0 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): my global_id is 124119 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc74003520 con 0x7fbc881360a0 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 337758396 0 0) 0x7fbc880a0430 con 0x7fbc880a54b0 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): my global_id is 124103 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc88142630 con 0x7fbc880a54b0 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 905167270 0 0) 0x7fbc880a0710 con 0x7fbc88135c80 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): my global_id is 124116 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc880a0430 con 0x7fbc88135c80 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3787780884 0 0) 0x7fbc88142630 con 0x7fbc880a54b0 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc880a0710 con 0x7fbc880a54b0 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2744919143 0 0) 0x7fbc880a0430 con 0x7fbc88135c80 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc88142630 con 0x7fbc88135c80 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 625345915 0 0) 0x7fbc74003520 con 0x7fbc881360a0 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc880a0430 con 0x7fbc881360a0 2024-09-17T15:49:58.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbc80001b80 con 0x7fbc880a54b0 2024-09-17T15:49:58.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:58.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:58.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): dump: 2024-09-17T15:49:58.703 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.703 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.703 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.703 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.703 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.703 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.703 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.703 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.704 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.704 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.704 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.704 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.704 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.704 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.704 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.704 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:49:58.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.681+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbc7c001a50 con 0x7fbc88135c80 2024-09-17T15:49:58.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:58.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:49:58.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient(hunting): dump: 2024-09-17T15:49:58.705 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.705 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.705 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.705 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.705 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.705 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.705 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.705 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.705 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.705 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.706 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.706 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.706 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.706 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.706 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.706 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbc78001a70 con 0x7fbc881360a0 2024-09-17T15:49:58.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:58.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:58.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient(hunting): dump: 2024-09-17T15:49:58.706 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.707 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.707 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.707 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.707 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.707 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.707 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.707 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.707 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.707 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.707 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.707 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.708 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.708 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.708 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.708 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2512278174 0 0) 0x7fbc88142630 con 0x7fbc88135c80 2024-09-17T15:49:58.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 >> v1:172.21.15.64:6790/0 conn(0x7fbc880a54b0 legacy=0x7fbc88141810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 >> v1:172.21.15.64:6789/0 conn(0x7fbc881360a0 legacy=0x7fbc881d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:49:58.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 1 monclient: found mon.d 2024-09-17T15:49:58.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:58.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:49:58.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc880a09f0 con 0x7fbc88135c80 2024-09-17T15:49:58.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient: _finish_auth 0 2024-09-17T15:49:58.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:49:58.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:58.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc9042a640 5 monclient: authenticate success, global_id 124116 2024-09-17T15:49:58.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc9042a640 10 monclient: _renew_subs 2024-09-17T15:49:58.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc9042a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:49:58.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc9042a640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc880a0cd0 con 0x7fbc88135c80 2024-09-17T15:49:58.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc9042a640 10 monclient: _renew_subs 2024-09-17T15:49:58.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc9042a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:49:58.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc9042a640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc8809f310 con 0x7fbc88135c80 2024-09-17T15:49:58.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc7c001bf0 con 0x7fbc88135c80 2024-09-17T15:49:58.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:58.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbc7c003be0 con 0x7fbc88135c80 2024-09-17T15:49:58.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc8c9c3640 4 set_mon_vals no callback set 2024-09-17T15:49:58.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:49:58.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.682+0000 7fbc86ffd640 10 monclient: dump: 2024-09-17T15:49:58.710 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.710 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.710 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.710 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.710 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.711 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.711 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.711 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.711 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.711 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.711 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.711 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.711 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.712 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.712 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.712 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc9042a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:58.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc9042a640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbc680065c0 con 0x7fbc68002860 2024-09-17T15:49:58.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fbc7c0104c0 con 0x7fbc88135c80 2024-09-17T15:49:58.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(193..193 src has 1..193) ==== 3506+0+0 (unknown 1845487835 0 0) 0x7fbc7c04d6b0 con 0x7fbc88135c80 2024-09-17T15:49:58.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2004159503 0 0) 0x7fbc680065c0 con 0x7fbc68002860 2024-09-17T15:49:58.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 10 monclient(hunting): my global_id is 124112 2024-09-17T15:49:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc88142630 con 0x7fbc68002860 2024-09-17T15:49:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4174798976 0 0) 0x7fbc88142630 con 0x7fbc68002860 2024-09-17T15:49:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc680065c0 con 0x7fbc68002860 2024-09-17T15:49:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbc80003eb0 con 0x7fbc68002860 2024-09-17T15:49:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc8e9c7640 10 monclient: get_auth_request con 0x7fbc74040c50 auth_method 0 2024-09-17T15:49:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 10 monclient: dump: 2024-09-17T15:49:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.713 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.714 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.714 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.714 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.714 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.714 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.714 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.714 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.714 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.714 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.714 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.714 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.715 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.715 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3948781394 0 0) 0x7fbc680065c0 con 0x7fbc68002860 2024-09-17T15:49:58.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.683+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc6800a160 con 0x7fbc68002860 2024-09-17T15:49:58.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.684+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbc800041a0 con 0x7fbc68002860 2024-09-17T15:49:58.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.684+0000 7fbc86ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:58.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.684+0000 7fbc86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:58.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.684+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 >> v1:172.21.15.64:6790/0 conn(0x7fbc68002860 legacy=0x7fbc68004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.684+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 >> v1:172.21.15.64:6790/0 conn(0x7fbc68002860 legacy=0x7fbc68004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:58.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc9042a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:58.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc9042a640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbc68009e50 con 0x7fbc68009110 2024-09-17T15:49:58.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4114320028 0 0) 0x7fbc68009e50 con 0x7fbc68009110 2024-09-17T15:49:58.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc86ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc86ffd640 10 monclient(hunting): my global_id is 124121 2024-09-17T15:49:58.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc680065c0 con 0x7fbc68009110 2024-09-17T15:49:58.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2373935222 0 0) 0x7fbc680065c0 con 0x7fbc68009110 2024-09-17T15:49:58.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc68009e50 con 0x7fbc68009110 2024-09-17T15:49:58.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fbc80005b00 con 0x7fbc68009110 2024-09-17T15:49:58.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:58.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.690+0000 7fbc86ffd640 10 monclient: dump: 2024-09-17T15:49:58.718 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.718 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.719 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.719 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.719 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.719 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.719 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.719 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.719 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.719 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.719 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.719 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.719 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.719 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.720 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4142345993 0 0) 0x7fbc68009e50 con 0x7fbc68009110 2024-09-17T15:49:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 --> v1:172.21.15.64:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbc6800c3f0 con 0x7fbc68009110 2024-09-17T15:49:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 3625743105) 0x7fbc80006f50 con 0x7fbc68009110 2024-09-17T15:49:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc86ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc86ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 >> v1:172.21.15.64:6790/0 conn(0x7fbc68009110 legacy=0x7fbc680095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc86ffd640 1 -- 172.21.15.64:0/1471222660 >> v1:172.21.15.64:6790/0 conn(0x7fbc68009110 legacy=0x7fbc680095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:58.721 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:49:58.721 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:49:58.721 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:49:58.721 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:49:58.721 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:49:58.721 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 5609, 2024-09-17T15:49:58.721 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:49:58.721 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:49:58.721 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:49:58.721 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:49:58.722 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:49:58.722 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:49:58.722 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:49:58.722 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:49:58.722 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:49:58.722 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:49:58.722 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:58.722 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:49:58.722 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:58.722 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:58.722 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:49:58.723 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:58.723 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:58.723 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:58.723 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:58.723 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:58.723 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:58.723 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:58.723 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:58.723 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:58.723 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:58.723 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:58.724 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:58.724 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:58.724 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:49:58.724 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:58.724 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:58.724 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:58.724 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:58.724 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:58.724 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:58.724 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:58.724 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:58.725 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:58.725 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:58.725 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:58.725 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.725 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:49:58.725 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:49:58.725 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:49:58.725 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:49:58.725 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:49:58.725 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:49:58.726 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:58.726 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:58.726 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:49:58.726 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:58.726 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:49:58.726 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:58.726 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:49:58.726 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:58.726 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:49:58.726 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:58.726 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:49:58.726 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:58.727 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:58.727 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:58.727 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:58.727 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:58.727 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:58.727 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:58.727 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:58.727 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:58.727 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:58.727 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:58.727 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:58.727 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:49:58.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.728 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:49:58.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.728 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:49:58.728 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:49:58.728 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.728 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.728 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.728 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:49:58.728 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.729 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.729 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.729 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:49:58.729 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:49:58.729 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.729 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.729 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.729 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:49:58.730 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:49:58.730 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.730 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.730 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.730 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:49:58.730 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.730 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.730 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.730 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:49:58.730 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:49:58.731 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.731 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.731 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.731 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:49:58.731 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:49:58.731 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.731 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.731 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.731 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:49:58.732 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.732 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.732 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:49:58.732 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:49:58.732 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.732 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.732 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.733 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:49:58.733 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:49:58.733 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.733 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.733 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.733 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:49:58.733 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.733 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.733 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:49:58.734 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:49:58.734 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.734 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.734 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.734 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:49:58.734 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:49:58.734 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.735 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.735 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.735 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:49:58.735 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.735 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.735 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:49:58.735 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:49:58.735 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.735 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.735 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.736 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:49:58.736 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:49:58.736 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.736 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.736 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.736 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:49:58.736 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.736 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.737 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:49:58.738 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.738 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.738 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.738 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:49:58.738 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:49:58.738 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.738 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.738 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.738 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:49:58.738 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.739 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.740 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:49:58.740 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:49:58.740 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.740 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.740 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.740 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:49:58.740 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.740 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.741 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:49:58.741 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:49:58.741 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.741 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.741 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.741 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.741 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:49:58.741 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:49:58.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.742 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:58.742 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:58.742 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:58.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.742 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:58.742 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:49:58.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.742 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:58.742 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:58.742 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:58.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.742 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.743 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:49:58.743 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:49:58.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc9042a640 1 -- 172.21.15.64:0/1471222660 >> v1:172.21.15.100:6812/4210963484 conn(0x7fbc74040c50 legacy=0x7fbc74043110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc9042a640 10 monclient: shutdown 2024-09-17T15:49:58.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc9042a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:58.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc9042a640 1 -- 172.21.15.64:0/1471222660 >> v1:172.21.15.64:6792/0 conn(0x7fbc88135c80 legacy=0x7fbc88141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc8e9c7640 1 -- 172.21.15.64:0/1471222660 reap_dead start 2024-09-17T15:49:58.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc8e9c7640 1 -- 172.21.15.64:0/1471222660 reap_dead start 2024-09-17T15:49:58.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc9042a640 1 -- 172.21.15.64:0/1471222660 shutdown_connections 2024-09-17T15:49:58.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc9042a640 1 -- 172.21.15.64:0/1471222660 >> 172.21.15.64:0/1471222660 conn(0x7fbc8806daa0 msgr2=0x7fbc880a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:58.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc9042a640 1 -- 172.21.15.64:0/1471222660 shutdown_connections 2024-09-17T15:49:58.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.691+0000 7fbc9042a640 1 -- 172.21.15.64:0/1471222660 wait complete. 2024-09-17T15:49:58.745 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:49:58.745 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-17T15:49:58.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:49:58.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.845+0000 7f2093b89640 10 monclient: get_monmap_and_config 2024-09-17T15:49:58.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.845+0000 7f2093b89640 10 monclient: build_initial_monmap 2024-09-17T15:49:58.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.846+0000 7f2093b89640 10 monclient: monmap: 2024-09-17T15:49:58.848 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:58.848 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.848 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:49:58.846855+0000 2024-09-17T15:49:58.848 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:49:58.846855+0000 2024-09-17T15:49:58.848 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:58.848 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.848 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:58.848 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:58.848 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:58.848 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:58.848 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:58.849 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:58.849 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:58.849 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:58.849 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:58.849 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.846+0000 7f2093b89640 1 Processor -- start 2024-09-17T15:49:58.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.846+0000 7f2093b89640 1 -- start start 2024-09-17T15:49:58.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.846+0000 7f2093b89640 10 monclient: init 2024-09-17T15:49:58.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2093b89640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:58.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2093b89640 10 monclient: _add_conns ranks=[6,8,4,7,1,2,5,0,3] 2024-09-17T15:49:58.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2093b89640 10 monclient(hunting): picked mon.noname-d con 0x7f208c0a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:49:58.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2093b89640 10 monclient(hunting): picked mon.noname-e con 0x7f208c135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:49:58.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2093b89640 10 monclient(hunting): picked mon.noname-c con 0x7f208c135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2093b89640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f208c09d6f0 con 0x7f208c135f70 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2093b89640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f208c09d9d0 con 0x7f208c0a5520 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2093b89640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f208c09dcb0 con 0x7f208c135b50 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2093b89640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2093b89640 10 monclient(hunting): authenticate will time out at 996.522370s 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2091124640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f208c135b50 0x7f208c13deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:58014/0 (socket says 172.21.15.64:58014) 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2091124640 1 -- 172.21.15.64:0/3555221704 learned_addr learned my addr 172.21.15.64:0/3555221704 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 1 -- 172.21.15.64:0/3555221704 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2798339142 0 0) 0x7f208c09d6f0 con 0x7f208c135f70 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 10 monclient(hunting): my global_id is 5737 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 1 -- 172.21.15.64:0/3555221704 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2074003520 con 0x7f208c135f70 2024-09-17T15:49:58.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 1 -- 172.21.15.64:0/3555221704 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 948533541 0 0) 0x7f208c09dcb0 con 0x7f208c135b50 2024-09-17T15:49:58.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 10 monclient(hunting): my global_id is 5633 2024-09-17T15:49:58.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 1 -- 172.21.15.64:0/3555221704 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f208c09d6f0 con 0x7f208c135b50 2024-09-17T15:49:58.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 1 -- 172.21.15.64:0/3555221704 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 668278621 0 0) 0x7f208c09d9d0 con 0x7f208c0a5520 2024-09-17T15:49:58.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 10 monclient(hunting): my global_id is 124125 2024-09-17T15:49:58.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 1 -- 172.21.15.64:0/3555221704 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f208c09dcb0 con 0x7f208c0a5520 2024-09-17T15:49:58.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.847+0000 7f2090923640 1 -- 172.21.15.64:0/3555221704 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2451529080 0 0) 0x7f208c09d6f0 con 0x7f208c135b50 2024-09-17T15:49:58.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 1 -- 172.21.15.64:0/3555221704 >> v1:172.21.15.64:6792/0 conn(0x7f208c0a5520 legacy=0x7f208c0a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 1 -- 172.21.15.64:0/3555221704 >> v1:172.21.15.64:6791/0 conn(0x7f208c135f70 legacy=0x7f208c1415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 10 monclient: _finish_hunting 0 2024-09-17T15:49:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 1 monclient: found mon.noname-e 2024-09-17T15:49:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:49:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 1 -- 172.21.15.64:0/3555221704 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f208c09ccd0 con 0x7f208c135b50 2024-09-17T15:49:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 10 monclient: _finish_auth 0 2024-09-17T15:49:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 10 monclient: _check_auth_tickets 2024-09-17T15:49:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 1 -- 172.21.15.64:0/3555221704 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2080001740 con 0x7f208c135b50 2024-09-17T15:49:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:49:58.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 10 monclient: dump: 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.861 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2093b89640 5 monclient: authenticate success, global_id 5633 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2093b89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 1 -- 172.21.15.64:0/3555221704 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20800018e0 con 0x7f208c135b50 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 1 -- 172.21.15.64:0/3555221704 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2080001bd0 con 0x7f208c135b50 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2093b89640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2093b89640 4 set_mon_vals no callback set 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2090923640 10 monclient: dump: 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.862 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.863 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2093b89640 10 monclient: shutdown 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2093b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2093b89640 1 -- 172.21.15.64:0/3555221704 >> v1:172.21.15.64:6793/0 conn(0x7f208c135b50 legacy=0x7f208c13deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2093b89640 1 -- 172.21.15.64:0/3555221704 shutdown_connections 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2093b89640 1 -- 172.21.15.64:0/3555221704 >> 172.21.15.64:0/3555221704 conn(0x7f208c06db30 msgr2=0x7f208c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2093b89640 1 -- 172.21.15.64:0/3555221704 shutdown_connections 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.848+0000 7f2093b89640 1 -- 172.21.15.64:0/3555221704 wait complete. 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.849+0000 7f2093b89640 10 monclient: build_initial_monmap 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.849+0000 7f2093b89640 10 monclient: monmap: 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:49:58.864 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.849+0000 7f2093b89640 1 Processor -- start 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.849+0000 7f2093b89640 1 -- start start 2024-09-17T15:49:58.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.849+0000 7f2093b89640 10 monclient: init 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f2093b89640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f2093b89640 10 monclient: _add_conns ranks=[8,0,3,2,1,4,7,5,6] 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f2093b89640 10 monclient(hunting): picked mon.noname-e con 0x7f208c0a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f2093b89640 10 monclient(hunting): picked mon.noname-a con 0x7f208c135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f2093b89640 10 monclient(hunting): picked mon.noname-g con 0x7f208c135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f2093b89640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f208c135060 con 0x7f208c135b50 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f2093b89640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f208c0a8d70 con 0x7f208c135f70 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f2093b89640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f208c1ce590 con 0x7f208c0a5520 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f2093b89640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f2093b89640 10 monclient(hunting): authenticate will time out at 996.525369s 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f2091925640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f208c0a5520 0x7f208c1cd770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:58020/0 (socket says 172.21.15.64:58020) 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f2091925640 1 -- 172.21.15.64:0/3895613736 learned_addr learned my addr 172.21.15.64:0/3895613736 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1545335702 0 0) 0x7f208c1ce590 con 0x7f208c0a5520 2024-09-17T15:49:58.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): my global_id is 5642 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2078003520 con 0x7f208c0a5520 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2803356725 0 0) 0x7f208c135060 con 0x7f208c135b50 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): my global_id is 124128 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f208c1ce590 con 0x7f208c135b50 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 956435863 0 0) 0x7f208c0a8d70 con 0x7f208c135f70 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): my global_id is 114780 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f208c135060 con 0x7f208c135f70 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 557360429 0 0) 0x7f2078003520 con 0x7f208c0a5520 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f208c0a8d70 con 0x7f208c0a5520 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2084001c10 con 0x7f208c0a5520 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): dump: 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.868 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1493401314 0 0) 0x7f208c1ce590 con 0x7f208c135b50 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.850+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2078003520 con 0x7f208c135b50 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2080001740 con 0x7f208c135b50 2024-09-17T15:49:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient(hunting): dump: 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.870 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1931906883 0 0) 0x7f208c135060 con 0x7f208c135f70 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f208c1ce590 con 0x7f208c135f70 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f207c001d70 con 0x7f208c135f70 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient(hunting): dump: 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.871 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.872 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.872 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.872 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.872 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.872 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.872 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.872 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.872 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.872 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.872 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.872 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.872 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1457053927 0 0) 0x7f208c0a8d70 con 0x7f208c0a5520 2024-09-17T15:49:58.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 >> v1:172.21.15.100:6790/0 conn(0x7f208c135f70 legacy=0x7f208c1d5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 >> v1:172.21.15.64:6789/0 conn(0x7f208c135b50 legacy=0x7f208c1cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:49:58.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 1 monclient: found mon.e 2024-09-17T15:49:58.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:58.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:58.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f208c09f040 con 0x7f208c0a5520 2024-09-17T15:49:58.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:49:58.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:49:58.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:58.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f2093b89640 5 monclient: authenticate success, global_id 5642 2024-09-17T15:49:58.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f2093b89640 10 monclient: _renew_subs 2024-09-17T15:49:58.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f2093b89640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f2093b89640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f208c09fe20 con 0x7f208c0a5520 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f2093b89640 10 monclient: _renew_subs 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f2093b89640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f2093b89640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f208c0a03e0 con 0x7f208c0a5520 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20840038a0 con 0x7f208c0a5520 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2084003e10 con 0x7f208c0a5520 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208bfff640 4 set_mon_vals no callback set 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.851+0000 7f208a7fc640 10 monclient: dump: 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.874 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f2084004030 con 0x7f208c0a5520 2024-09-17T15:49:58.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f2093b89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:58.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f2093b89640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f206c0065c0 con 0x7f206c002860 2024-09-17T15:49:58.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(193..193 src has 1..193) ==== 3506+0+0 (unknown 1845487835 0 0) 0x7f208404e9b0 con 0x7f208c0a5520 2024-09-17T15:49:58.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 312387298 0 0) 0x7f206c0065c0 con 0x7f206c002860 2024-09-17T15:49:58.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 10 monclient(hunting): my global_id is 5746 2024-09-17T15:49:58.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f208c0a8d70 con 0x7f206c002860 2024-09-17T15:49:58.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f2092126640 10 monclient: get_auth_request con 0x7f2078040cf0 auth_method 0 2024-09-17T15:49:58.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 862070986 0 0) 0x7f208c0a8d70 con 0x7f206c002860 2024-09-17T15:49:58.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f206c0065c0 con 0x7f206c002860 2024-09-17T15:49:58.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f20800018e0 con 0x7f206c002860 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 10 monclient: dump: 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.877 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.878 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4294230622 0 0) 0x7f206c0065c0 con 0x7f206c002860 2024-09-17T15:49:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.852+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f206c00a160 con 0x7f206c002860 2024-09-17T15:49:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.853+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2080005080 con 0x7f206c002860 2024-09-17T15:49:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.853+0000 7f208a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.853+0000 7f208a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.853+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 >> v1:172.21.15.64:6791/0 conn(0x7f206c002860 legacy=0x7f206c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.853+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 >> v1:172.21.15.64:6791/0 conn(0x7f206c002860 legacy=0x7f206c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:58.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f2093b89640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:58.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f2093b89640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f206c009e50 con 0x7f206c009110 2024-09-17T15:49:58.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2157944259 0 0) 0x7f206c009e50 con 0x7f206c009110 2024-09-17T15:49:58.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f208a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:49:58.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f208a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f208a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f208a7fc640 10 monclient(hunting): my global_id is 5755 2024-09-17T15:49:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f206c0065c0 con 0x7f206c009110 2024-09-17T15:49:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1772697816 0 0) 0x7f206c0065c0 con 0x7f206c009110 2024-09-17T15:49:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f206c009e50 con 0x7f206c009110 2024-09-17T15:49:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f20800058a0 con 0x7f206c009110 2024-09-17T15:49:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f208a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f208a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.859+0000 7f208a7fc640 10 monclient: dump: 2024-09-17T15:49:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:58.881 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2329875993 0 0) 0x7f206c009e50 con 0x7f206c009110 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 --> v1:172.21.15.64:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f206c00c3f0 con 0x7f206c009110 2024-09-17T15:49:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1602957321) 0x7f2080006c40 con 0x7f206c009110 2024-09-17T15:49:58.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f208a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:58.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f208a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:58.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 >> v1:172.21.15.64:6791/0 conn(0x7f206c009110 legacy=0x7f206c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f208a7fc640 1 -- 172.21.15.64:0/3895613736 >> v1:172.21.15.64:6791/0 conn(0x7f206c009110 legacy=0x7f206c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:58.883 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:49:58.883 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:49:58.883 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:49:58.883 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:49:58.883 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:49:58.883 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 255517, 2024-09-17T15:49:58.884 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:49:58.884 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:49:58.884 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:49:58.884 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:49:58.884 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:49:58.884 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:49:58.884 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:49:58.884 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:49:58.884 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:49:58.884 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:49:58.884 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:58.884 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:49:58.884 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:58.885 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:58.886 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.887 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:49:58.887 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:49:58.887 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:49:58.887 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:49:58.887 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:49:58.887 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:49:58.887 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:58.887 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:58.887 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:49:58.888 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:58.888 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:49:58.888 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:58.888 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:49:58.888 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:58.888 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:49:58.888 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:58.888 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:49:58.888 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:58.888 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:58.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:58.888 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:58.888 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:49:58.889 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.891 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:49:58.891 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:49:58.891 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.891 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.891 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.891 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:49:58.891 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.891 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.891 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:49:58.892 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.893 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:49:58.895 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.896 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.897 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:49:58.898 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.899 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:49:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:49:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:49:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:58.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 4 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:49:58.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:58.903 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:58.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:58.903 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:49:58.903 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:49:58.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f2093b89640 1 -- 172.21.15.64:0/3895613736 >> v1:172.21.15.100:6812/4210963484 conn(0x7f2078040cf0 legacy=0x7f20780431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f2093b89640 10 monclient: shutdown 2024-09-17T15:49:58.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f2093b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:58.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f2093b89640 1 -- 172.21.15.64:0/3895613736 >> v1:172.21.15.64:6793/0 conn(0x7f208c0a5520 legacy=0x7f208c1cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f2092126640 1 -- 172.21.15.64:0/3895613736 reap_dead start 2024-09-17T15:49:58.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f2092126640 1 -- 172.21.15.64:0/3895613736 reap_dead start 2024-09-17T15:49:58.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f2093b89640 1 -- 172.21.15.64:0/3895613736 shutdown_connections 2024-09-17T15:49:58.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f2093b89640 1 -- 172.21.15.64:0/3895613736 >> 172.21.15.64:0/3895613736 conn(0x7f208c06db30 msgr2=0x7f208c0a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:58.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f2093b89640 1 -- 172.21.15.64:0/3895613736 shutdown_connections 2024-09-17T15:49:58.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:58.860+0000 7f2093b89640 1 -- 172.21.15.64:0/3895613736 wait complete. 2024-09-17T15:49:58.908 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:49:58.908 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-17T15:49:58.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-17T15:49:59.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.008+0000 7fc7e44fe640 10 monclient: get_monmap_and_config 2024-09-17T15:49:59.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.008+0000 7fc7e44fe640 10 monclient: build_initial_monmap 2024-09-17T15:49:59.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.008+0000 7fc7e44fe640 10 monclient: monmap: 2024-09-17T15:49:59.011 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:59.011 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.011 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:49:59.009722+0000 2024-09-17T15:49:59.011 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:49:59.009722+0000 2024-09-17T15:49:59.011 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:59.011 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.011 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:59.011 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:59.011 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:59.011 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:59.012 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:59.012 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:59.012 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:59.012 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:59.012 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:59.012 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.009+0000 7fc7e44fe640 1 Processor -- start 2024-09-17T15:49:59.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.009+0000 7fc7e44fe640 1 -- start start 2024-09-17T15:49:59.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.009+0000 7fc7e44fe640 10 monclient: init 2024-09-17T15:49:59.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:59.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 10 monclient: _add_conns ranks=[1,5,2,6,4,8,7,3,0] 2024-09-17T15:49:59.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 10 monclient(hunting): picked mon.noname-f con 0x7fc7dc0a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:49:59.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 10 monclient(hunting): picked mon.noname-h con 0x7fc7dc135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:49:59.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 10 monclient(hunting): picked mon.noname-b con 0x7fc7dc135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:49:59.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc09d6f0 con 0x7fc7dc0a5520 2024-09-17T15:49:59.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc09d9d0 con 0x7fc7dc135f70 2024-09-17T15:49:59.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc09dcb0 con 0x7fc7dc135b50 2024-09-17T15:49:59.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:59.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 10 monclient(hunting): authenticate will time out at 996.685190s 2024-09-17T15:49:59.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e2a9b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc7dc135f70 0x7fc7dc1415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46910/0 (socket says 172.21.15.64:46910) 2024-09-17T15:49:59.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e2a9b640 1 -- 172.21.15.64:0/2818812517 learned_addr learned my addr 172.21.15.64:0/2818812517 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:59.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 -- 172.21.15.64:0/2818812517 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1388292922 0 0) 0x7fc7dc09d9d0 con 0x7fc7dc135f70 2024-09-17T15:49:59.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient(hunting): my global_id is 124130 2024-09-17T15:49:59.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 -- 172.21.15.64:0/2818812517 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7c4003520 con 0x7fc7dc135f70 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 -- 172.21.15.64:0/2818812517 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1801852337 0 0) 0x7fc7dc09d6f0 con 0x7fc7dc0a5520 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient(hunting): my global_id is 5824 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 -- 172.21.15.64:0/2818812517 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7dc09d9d0 con 0x7fc7dc0a5520 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 -- 172.21.15.64:0/2818812517 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 785667122 0 0) 0x7fc7dc09dcb0 con 0x7fc7dc135b50 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient(hunting): my global_id is 124142 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 -- 172.21.15.64:0/2818812517 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7dc09d6f0 con 0x7fc7dc135b50 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 -- 172.21.15.64:0/2818812517 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2868871679 0 0) 0x7fc7c4003520 con 0x7fc7dc135f70 2024-09-17T15:49:59.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 -- 172.21.15.64:0/2818812517 >> v1:172.21.15.100:6791/0 conn(0x7fc7dc135b50 legacy=0x7fc7dc13deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 -- 172.21.15.64:0/2818812517 >> v1:172.21.15.100:6789/0 conn(0x7fc7dc0a5520 legacy=0x7fc7dc0a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient: _finish_hunting 0 2024-09-17T15:49:59.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 monclient: found mon.noname-b 2024-09-17T15:49:59.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:59.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:49:59.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 -- 172.21.15.64:0/2818812517 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7dc09ccd0 con 0x7fc7dc135f70 2024-09-17T15:49:59.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient: _finish_auth 0 2024-09-17T15:49:59.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient: _check_auth_tickets 2024-09-17T15:49:59.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:59.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 -- 172.21.15.64:0/2818812517 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc7cc001bb0 con 0x7fc7dc135f70 2024-09-17T15:49:59.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 5 monclient: authenticate success, global_id 124130 2024-09-17T15:49:59.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:59.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:59.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient: dump: 2024-09-17T15:49:59.025 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.025 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.025 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.025 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.025 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.025 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.025 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.025 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.025 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.025 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.025 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.026 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.026 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.026 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.026 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.026 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:59.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 -- 172.21.15.64:0/2818812517 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc7cc001dd0 con 0x7fc7dc135f70 2024-09-17T15:49:59.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:59.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 1 -- 172.21.15.64:0/2818812517 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc7cc004f60 con 0x7fc7dc135f70 2024-09-17T15:49:59.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:59.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e44fe640 4 set_mon_vals no callback set 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.010+0000 7fc7e1298640 10 monclient: dump: 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.027 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.028 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.028 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.028 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.028 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.028 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.011+0000 7fc7e44fe640 10 monclient: shutdown 2024-09-17T15:49:59.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.011+0000 7fc7e44fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:59.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.011+0000 7fc7e44fe640 1 -- 172.21.15.64:0/2818812517 >> v1:172.21.15.64:6790/0 conn(0x7fc7dc135f70 legacy=0x7fc7dc1415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.011+0000 7fc7e44fe640 1 -- 172.21.15.64:0/2818812517 shutdown_connections 2024-09-17T15:49:59.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.011+0000 7fc7e44fe640 1 -- 172.21.15.64:0/2818812517 >> 172.21.15.64:0/2818812517 conn(0x7fc7dc06db30 msgr2=0x7fc7dc0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:59.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.011+0000 7fc7e44fe640 1 -- 172.21.15.64:0/2818812517 shutdown_connections 2024-09-17T15:49:59.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.011+0000 7fc7e44fe640 1 -- 172.21.15.64:0/2818812517 wait complete. 2024-09-17T15:49:59.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 10 monclient: build_initial_monmap 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 10 monclient: monmap: 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:59.029 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 1 Processor -- start 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 1 -- start start 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 10 monclient: init 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 10 monclient: _add_conns ranks=[8,4,5,0,7,1,6,2,3] 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 10 monclient(hunting): picked mon.noname-e con 0x7fc7dc0a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 10 monclient(hunting): picked mon.noname-c con 0x7fc7dc135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 10 monclient(hunting): picked mon.noname-h con 0x7fc7dc135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc13aa00 con 0x7fc7dc135b50 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc13ace0 con 0x7fc7dc135f70 2024-09-17T15:49:59.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc13a720 con 0x7fc7dc0a5520 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.012+0000 7fc7e44fe640 10 monclient(hunting): authenticate will time out at 996.687996s 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7e229a640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fc7dc0a5520 0x7fc7dc137560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:58034/0 (socket says 172.21.15.64:58034) 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7e229a640 1 -- 172.21.15.64:0/151043269 learned_addr learned my addr 172.21.15.64:0/151043269 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432670998 0 0) 0x7fc7dc09d6f0 con 0x7fc7dc135b50 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): my global_id is 5764 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7c8003550 con 0x7fc7dc135b50 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2997676673 0 0) 0x7fc7dc13a720 con 0x7fc7dc0a5520 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): my global_id is 5651 2024-09-17T15:49:59.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7dc09d6f0 con 0x7fc7dc0a5520 2024-09-17T15:49:59.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 30984540 0 0) 0x7fc7dc13ace0 con 0x7fc7dc135f70 2024-09-17T15:49:59.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): my global_id is 124151 2024-09-17T15:49:59.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7dc13a720 con 0x7fc7dc135f70 2024-09-17T15:49:59.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 225367394 0 0) 0x7fc7dc09d6f0 con 0x7fc7dc0a5520 2024-09-17T15:49:59.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7dc13ace0 con 0x7fc7dc0a5520 2024-09-17T15:49:59.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1168796321 0 0) 0x7fc7c8003550 con 0x7fc7dc135b50 2024-09-17T15:49:59.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7dc09d6f0 con 0x7fc7dc135b50 2024-09-17T15:49:59.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc7d4001b10 con 0x7fc7dc0a5520 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): dump: 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.033 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.034 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.034 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.034 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.034 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.034 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:49:59.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc7d0001a10 con 0x7fc7dc135b50 2024-09-17T15:49:59.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:59.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): dump: 2024-09-17T15:49:59.034 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.034 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.034 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.035 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.035 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.035 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.035 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.035 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.035 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.035 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.035 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.035 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.035 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.035 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.035 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.035 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 453686308 0 0) 0x7fc7dc13a720 con 0x7fc7dc135f70 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7c8003550 con 0x7fc7dc135f70 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc7cc0013c0 con 0x7fc7dc135f70 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient(hunting): dump: 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.036 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.037 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.037 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.037 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.037 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.037 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.037 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.037 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.037 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4140555718 0 0) 0x7fc7dc13ace0 con 0x7fc7dc0a5520 2024-09-17T15:49:59.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 >> v1:172.21.15.100:6791/0 conn(0x7fc7dc135f70 legacy=0x7fc7dc1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 >> v1:172.21.15.64:6791/0 conn(0x7fc7dc135b50 legacy=0x7fc7dc137c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient: _finish_hunting 0 2024-09-17T15:49:59.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 monclient: found mon.e 2024-09-17T15:49:59.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:59.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7dc1397c0 con 0x7fc7dc0a5520 2024-09-17T15:49:59.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient: _finish_auth 0 2024-09-17T15:49:59.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient: _check_auth_tickets 2024-09-17T15:49:59.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:59.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7e44fe640 5 monclient: authenticate success, global_id 5651 2024-09-17T15:49:59.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7e44fe640 10 monclient: _renew_subs 2024-09-17T15:49:59.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7e44fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7e44fe640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc7dc13a2d0 con 0x7fc7dc0a5520 2024-09-17T15:49:59.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7e44fe640 10 monclient: _renew_subs 2024-09-17T15:49:59.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7e44fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc7d40038c0 con 0x7fc7dc0a5520 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7e44fe640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fc7dc138560 con 0x7fc7dc0a5520 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc7d4003e30 con 0x7fc7dc0a5520 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7daffd640 10 monclient: dump: 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.039 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.013+0000 7fc7e0a97640 4 set_mon_vals no callback set 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.014+0000 7fc7e44fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.014+0000 7fc7e44fe640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc7dc0a63e0 con 0x7fc7dc142f10 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.014+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fc7d4004050 con 0x7fc7dc0a5520 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(193..193 src has 1..193) ==== 3506+0+0 (unknown 1845487835 0 0) 0x7fc7d404ead0 con 0x7fc7dc0a5520 2024-09-17T15:49:59.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1301086302 0 0) 0x7fc7dc0a63e0 con 0x7fc7dc142f10 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 10 monclient(hunting): my global_id is 124134 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7dc13ace0 con 0x7fc7dc142f10 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7e2a9b640 10 monclient: get_auth_request con 0x7fc7c8040c80 auth_method 0 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1275506267 0 0) 0x7fc7dc13ace0 con 0x7fc7dc142f10 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7dc0a63e0 con 0x7fc7dc142f10 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc7d0001bb0 con 0x7fc7dc142f10 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 10 monclient: dump: 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.041 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.042 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3846903040 0 0) 0x7fc7dc0a63e0 con 0x7fc7dc142f10 2024-09-17T15:49:59.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc7dc0008d0 con 0x7fc7dc142f10 2024-09-17T15:49:59.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc7d0004040 con 0x7fc7dc142f10 2024-09-17T15:49:59.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:59.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:59.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 >> v1:172.21.15.64:6792/0 conn(0x7fc7dc142f10 legacy=0x7fc7dc063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.015+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 >> v1:172.21.15.64:6792/0 conn(0x7fc7dc142f10 legacy=0x7fc7dc063d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:59.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7e44fe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:59.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7e44fe640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc7dc0a1ba0 con 0x7fc7dc02e5c0 2024-09-17T15:49:59.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1263224066 0 0) 0x7fc7dc0a1ba0 con 0x7fc7dc02e5c0 2024-09-17T15:49:59.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 10 monclient(hunting): my global_id is 124143 2024-09-17T15:49:59.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7dc0a63e0 con 0x7fc7dc02e5c0 2024-09-17T15:49:59.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3903510496 0 0) 0x7fc7dc0a63e0 con 0x7fc7dc02e5c0 2024-09-17T15:49:59.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7dc0a1ba0 con 0x7fc7dc02e5c0 2024-09-17T15:49:59.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc7d00047a0 con 0x7fc7dc02e5c0 2024-09-17T15:49:59.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:49:59.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 10 monclient: dump: 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.046 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.047 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1071779536 0 0) 0x7fc7dc0a1ba0 con 0x7fc7dc02e5c0 2024-09-17T15:49:59.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 --> v1:172.21.15.64:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc7dc02e9a0 con 0x7fc7dc02e5c0 2024-09-17T15:49:59.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 3508902995) 0x7fc7d0005ac0 con 0x7fc7dc02e5c0 2024-09-17T15:49:59.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:59.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:59.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 >> v1:172.21.15.64:6792/0 conn(0x7fc7dc02e5c0 legacy=0x7fc7dc0a1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.022+0000 7fc7daffd640 1 -- 172.21.15.64:0/151043269 >> v1:172.21.15.64:6792/0 conn(0x7fc7dc02e5c0 legacy=0x7fc7dc0a1490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:59.047 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:49:59.047 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:49:59.048 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:49:59.048 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:49:59.048 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:49:59.048 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 5940, 2024-09-17T15:49:59.048 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:49:59.048 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:49:59.048 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:49:59.048 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:49:59.048 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:49:59.048 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:49:59.048 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:49:59.048 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:49:59.048 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:49:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:49:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:49:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:49:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.050 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.051 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.051 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.051 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.051 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:49:59.051 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:49:59.051 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:49:59.051 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:49:59.051 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:49:59.051 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:49:59.051 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:59.051 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:49:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:49:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:49:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:49:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:49:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:49:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.054 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.055 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:49:59.055 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:49:59.055 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.055 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.055 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:49:59.055 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.055 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.056 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:49:59.056 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:49:59.056 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.056 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.056 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.056 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:49:59.056 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:49:59.056 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.056 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:49:59.057 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.058 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.059 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:49:59.060 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.061 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:49:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.064 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.064 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.064 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:49:59.064 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:49:59.064 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.064 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.064 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:49:59.064 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.064 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.065 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.066 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:59.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.066 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:49:59.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.066 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.066 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.066 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:59.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.066 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.066 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:49:59.066 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:49:59.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.023+0000 7fc7e44fe640 1 -- 172.21.15.64:0/151043269 >> v1:172.21.15.100:6812/4210963484 conn(0x7fc7c8040c80 legacy=0x7fc7c8043140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.023+0000 7fc7e44fe640 10 monclient: shutdown 2024-09-17T15:49:59.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.023+0000 7fc7e44fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:59.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.023+0000 7fc7e44fe640 1 -- 172.21.15.64:0/151043269 >> v1:172.21.15.64:6793/0 conn(0x7fc7dc0a5520 legacy=0x7fc7dc137560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.023+0000 7fc7e2a9b640 1 -- 172.21.15.64:0/151043269 reap_dead start 2024-09-17T15:49:59.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.023+0000 7fc7e2a9b640 1 -- 172.21.15.64:0/151043269 reap_dead start 2024-09-17T15:49:59.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.023+0000 7fc7e44fe640 1 -- 172.21.15.64:0/151043269 shutdown_connections 2024-09-17T15:49:59.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.023+0000 7fc7e44fe640 1 -- 172.21.15.64:0/151043269 >> 172.21.15.64:0/151043269 conn(0x7fc7dc06db30 msgr2=0x7fc7dc0a4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:59.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.023+0000 7fc7e44fe640 1 -- 172.21.15.64:0/151043269 shutdown_connections 2024-09-17T15:49:59.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.023+0000 7fc7e44fe640 1 -- 172.21.15.64:0/151043269 wait complete. 2024-09-17T15:49:59.069 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:49:59.069 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-17T15:49:59.069 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-17T15:49:59.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.173+0000 7f6ff9a01640 10 monclient: get_monmap_and_config 2024-09-17T15:49:59.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.173+0000 7f6ff9a01640 10 monclient: build_initial_monmap 2024-09-17T15:49:59.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.173+0000 7f6ff9a01640 10 monclient: monmap: 2024-09-17T15:49:59.175 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:59.175 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.175 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:49:59.174131+0000 2024-09-17T15:49:59.175 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:49:59.174131+0000 2024-09-17T15:49:59.175 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:59.175 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.175 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:59.175 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:59.176 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:59.176 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:59.176 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:59.176 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:59.176 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:59.176 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:59.176 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:59.176 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff9a01640 1 Processor -- start 2024-09-17T15:49:59.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff9a01640 1 -- start start 2024-09-17T15:49:59.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff9a01640 10 monclient: init 2024-09-17T15:49:59.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff9a01640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:59.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff9a01640 10 monclient: _add_conns ranks=[1,5,2,6,0,4,7,8,3] 2024-09-17T15:49:59.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff9a01640 10 monclient(hunting): picked mon.noname-f con 0x7f6ff40a54d0 addr v1:172.21.15.100:6789/0 2024-09-17T15:49:59.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff9a01640 10 monclient(hunting): picked mon.noname-h con 0x7f6ff4135f40 addr v1:172.21.15.100:6791/0 2024-09-17T15:49:59.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff9a01640 10 monclient(hunting): picked mon.noname-b con 0x7f6ff413f470 addr v1:172.21.15.64:6790/0 2024-09-17T15:49:59.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff9a01640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff409d6a0 con 0x7f6ff40a54d0 2024-09-17T15:49:59.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff9a01640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff409d980 con 0x7f6ff413f470 2024-09-17T15:49:59.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff9a01640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff409dc60 con 0x7f6ff4135f40 2024-09-17T15:49:59.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff9a01640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:59.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff9a01640 10 monclient(hunting): authenticate will time out at 996.849620s 2024-09-17T15:49:59.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff37fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f6ff413f470 0x7f6ff4141860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46924/0 (socket says 172.21.15.64:46924) 2024-09-17T15:49:59.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.174+0000 7f6ff37fe640 1 -- 172.21.15.64:0/934771656 learned_addr learned my addr 172.21.15.64:0/934771656 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:59.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 -- 172.21.15.64:0/934771656 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102426725 0 0) 0x7f6ff409d980 con 0x7f6ff413f470 2024-09-17T15:49:59.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient(hunting): my global_id is 124139 2024-09-17T15:49:59.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 -- 172.21.15.64:0/934771656 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe0003520 con 0x7f6ff413f470 2024-09-17T15:49:59.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 -- 172.21.15.64:0/934771656 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 117309351 0 0) 0x7f6ff409d6a0 con 0x7f6ff40a54d0 2024-09-17T15:49:59.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient(hunting): my global_id is 5833 2024-09-17T15:49:59.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 -- 172.21.15.64:0/934771656 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff409d980 con 0x7f6ff40a54d0 2024-09-17T15:49:59.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 -- 172.21.15.64:0/934771656 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2985920919 0 0) 0x7f6ff409dc60 con 0x7f6ff4135f40 2024-09-17T15:49:59.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient(hunting): my global_id is 124160 2024-09-17T15:49:59.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 -- 172.21.15.64:0/934771656 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff409d6a0 con 0x7f6ff4135f40 2024-09-17T15:49:59.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 -- 172.21.15.64:0/934771656 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2306211311 0 0) 0x7f6fe0003520 con 0x7f6ff413f470 2024-09-17T15:49:59.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 -- 172.21.15.64:0/934771656 >> v1:172.21.15.100:6791/0 conn(0x7f6ff4135f40 legacy=0x7f6ff413dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 -- 172.21.15.64:0/934771656 >> v1:172.21.15.100:6789/0 conn(0x7f6ff40a54d0 legacy=0x7f6ff40a58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient: _finish_hunting 0 2024-09-17T15:49:59.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 monclient: found mon.noname-b 2024-09-17T15:49:59.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:59.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:49:59.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 -- 172.21.15.64:0/934771656 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff409df40 con 0x7f6ff413f470 2024-09-17T15:49:59.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient: _finish_auth 0 2024-09-17T15:49:59.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:49:59.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:59.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff9a01640 5 monclient: authenticate success, global_id 124139 2024-09-17T15:49:59.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 -- 172.21.15.64:0/934771656 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6fec001bb0 con 0x7f6ff413f470 2024-09-17T15:49:59.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff9a01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:59.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:59.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient: dump: 2024-09-17T15:49:59.190 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.190 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.190 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.190 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.190 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.190 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.190 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.190 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.190 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.190 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.190 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.191 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.191 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.191 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.191 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.191 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff9a01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:59.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 -- 172.21.15.64:0/934771656 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6fec001d50 con 0x7f6ff413f470 2024-09-17T15:49:59.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:59.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff9a01640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:59.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 1 -- 172.21.15.64:0/934771656 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6fec004f60 con 0x7f6ff413f470 2024-09-17T15:49:59.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff9a01640 4 set_mon_vals no callback set 2024-09-17T15:49:59.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:59.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff27fc640 10 monclient: dump: 2024-09-17T15:49:59.192 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.192 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.192 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.192 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.192 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.192 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.192 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.193 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.193 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.193 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.193 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.193 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.193 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.193 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.193 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.193 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff9a01640 10 monclient: shutdown 2024-09-17T15:49:59.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff9a01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:59.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff9a01640 1 -- 172.21.15.64:0/934771656 >> v1:172.21.15.64:6790/0 conn(0x7f6ff413f470 legacy=0x7f6ff4141860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff9a01640 1 -- 172.21.15.64:0/934771656 shutdown_connections 2024-09-17T15:49:59.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff9a01640 1 -- 172.21.15.64:0/934771656 >> 172.21.15.64:0/934771656 conn(0x7f6ff406d9a0 msgr2=0x7f6ff40ab8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:59.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.175+0000 7f6ff9a01640 1 -- 172.21.15.64:0/934771656 shutdown_connections 2024-09-17T15:49:59.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.176+0000 7f6ff9a01640 1 -- 172.21.15.64:0/934771656 wait complete. 2024-09-17T15:49:59.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.176+0000 7f6ff9a01640 10 monclient: build_initial_monmap 2024-09-17T15:49:59.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.176+0000 7f6ff9a01640 10 monclient: monmap: 2024-09-17T15:49:59.194 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:59.194 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:59.194 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:49:59.194 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:49:59.195 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:59.195 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:49:59.195 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:59.195 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:59.195 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:59.195 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:59.195 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:59.195 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:59.195 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:59.195 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:59.195 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:59.196 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6ff9a01640 1 Processor -- start 2024-09-17T15:49:59.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6ff9a01640 1 -- start start 2024-09-17T15:49:59.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6ff9a01640 10 monclient: init 2024-09-17T15:49:59.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6ff9a01640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:59.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6ff9a01640 10 monclient: _add_conns ranks=[3,8,5,6,0,7,2,4,1] 2024-09-17T15:49:59.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6ff9a01640 10 monclient(hunting): picked mon.noname-g con 0x7f6ff40a54d0 addr v1:172.21.15.100:6790/0 2024-09-17T15:49:59.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6ff9a01640 10 monclient(hunting): picked mon.noname-e con 0x7f6ff4135f40 addr v1:172.21.15.64:6793/0 2024-09-17T15:49:59.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6ff9a01640 10 monclient(hunting): picked mon.noname-h con 0x7f6ff413f470 addr v1:172.21.15.100:6791/0 2024-09-17T15:49:59.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6ff9a01640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff413d680 con 0x7f6ff40a54d0 2024-09-17T15:49:59.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6ff9a01640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff40a8d20 con 0x7f6ff413f470 2024-09-17T15:49:59.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6ff9a01640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff4134e30 con 0x7f6ff4135f40 2024-09-17T15:49:59.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6ff9a01640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:59.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6ff9a01640 10 monclient(hunting): authenticate will time out at 996.852645s 2024-09-17T15:49:59.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6febfff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f6ff4135f40 0x7f6ff40a21a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:58036/0 (socket says 172.21.15.64:58036) 2024-09-17T15:49:59.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.177+0000 7f6febfff640 1 -- 172.21.15.64:0/1095682377 learned_addr learned my addr 172.21.15.64:0/1095682377 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:59.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2573010472 0 0) 0x7f6ff4134e30 con 0x7f6ff4135f40 2024-09-17T15:49:59.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): my global_id is 5660 2024-09-17T15:49:59.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd4003520 con 0x7f6ff4135f40 2024-09-17T15:49:59.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1664652297 0 0) 0x7f6ff40a8d20 con 0x7f6ff413f470 2024-09-17T15:49:59.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): my global_id is 124169 2024-09-17T15:49:59.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff4134e30 con 0x7f6ff413f470 2024-09-17T15:49:59.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2478662155 0 0) 0x7f6ff413d680 con 0x7f6ff40a54d0 2024-09-17T15:49:59.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): my global_id is 114789 2024-09-17T15:49:59.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff40a8d20 con 0x7f6ff40a54d0 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1127240193 0 0) 0x7f6fd4003520 con 0x7f6ff4135f40 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ff413d680 con 0x7f6ff4135f40 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6fdc001bb0 con 0x7f6ff4135f40 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): dump: 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.199 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.200 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.200 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.200 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.200 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.200 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.200 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.200 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.200 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:49:59.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 902152282 0 0) 0x7f6ff4134e30 con 0x7f6ff413f470 2024-09-17T15:49:59.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fd4003520 con 0x7f6ff413f470 2024-09-17T15:49:59.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6fec001a70 con 0x7f6ff413f470 2024-09-17T15:49:59.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): dump: 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.201 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.202 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.202 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.202 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.202 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 821718459 0 0) 0x7f6ff40a8d20 con 0x7f6ff40a54d0 2024-09-17T15:49:59.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ff4134e30 con 0x7f6ff40a54d0 2024-09-17T15:49:59.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6fe4001a50 con 0x7f6ff40a54d0 2024-09-17T15:49:59.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:49:59.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient(hunting): dump: 2024-09-17T15:49:59.202 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.202 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.202 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1959372369 0 0) 0x7f6ff413d680 con 0x7f6ff4135f40 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 >> v1:172.21.15.100:6791/0 conn(0x7f6ff413f470 legacy=0x7f6ff41d5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 >> v1:172.21.15.100:6790/0 conn(0x7f6ff40a54d0 legacy=0x7f6ff40a1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient: _finish_hunting 0 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 monclient: found mon.e 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff40a28b0 con 0x7f6ff4135f40 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient: _finish_auth 0 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff9a01640 5 monclient: authenticate success, global_id 5660 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff9a01640 10 monclient: _renew_subs 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff9a01640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff9a01640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff41369b0 con 0x7f6ff4135f40 2024-09-17T15:49:59.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff9a01640 10 monclient: _renew_subs 2024-09-17T15:49:59.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff9a01640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff9a01640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f6ff413e1e0 con 0x7f6ff4135f40 2024-09-17T15:49:59.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6fdc0038c0 con 0x7f6ff4135f40 2024-09-17T15:49:59.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:59.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6fdc003e30 con 0x7f6ff4135f40 2024-09-17T15:49:59.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff1ffb640 4 set_mon_vals no callback set 2024-09-17T15:49:59.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:49:59.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.178+0000 7f6ff0ff9640 10 monclient: dump: 2024-09-17T15:49:59.205 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.205 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.205 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff9a01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff9a01640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fd80065c0 con 0x7f6fd8002860 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f6fdc003fd0 con 0x7f6ff4135f40 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(193..193 src has 1..193) ==== 3506+0+0 (unknown 1845487835 0 0) 0x7f6fdc04eba0 con 0x7f6ff4135f40 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2805778491 0 0) 0x7f6fd80065c0 con 0x7f6fd8002860 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff0ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff0ff9640 10 monclient(hunting): my global_id is 5669 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff413d680 con 0x7f6fd8002860 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff37fe640 10 monclient: get_auth_request con 0x7f6fd4040da0 auth_method 0 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1535067293 0 0) 0x7f6ff413d680 con 0x7f6fd8002860 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fd80065c0 con 0x7f6fd8002860 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.179+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6fe4003930 con 0x7f6fd8002860 2024-09-17T15:49:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.180+0000 7f6ff0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.180+0000 7f6ff0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:49:59.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.180+0000 7f6ff0ff9640 10 monclient: dump: 2024-09-17T15:49:59.208 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.208 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.208 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.208 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.208 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.208 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.208 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.208 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.208 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.208 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.209 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.209 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.209 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.209 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.209 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.209 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.180+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1596943031 0 0) 0x7f6fd80065c0 con 0x7f6fd8002860 2024-09-17T15:49:59.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.180+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.64:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6fd800a160 con 0x7f6fd8002860 2024-09-17T15:49:59.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.180+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6fe4005b30 con 0x7f6fd8002860 2024-09-17T15:49:59.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.180+0000 7f6ff0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:59.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.180+0000 7f6ff0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:59.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.180+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 >> v1:172.21.15.64:6793/0 conn(0x7f6fd8002860 legacy=0x7f6fd8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.180+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 >> v1:172.21.15.64:6793/0 conn(0x7f6fd8002860 legacy=0x7f6fd8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:59.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.186+0000 7f6ff9a01640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:59.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.186+0000 7f6ff9a01640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fd8009e20 con 0x7f6fd80090e0 2024-09-17T15:49:59.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.186+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2294049430 0 0) 0x7f6fd8009e20 con 0x7f6fd80090e0 2024-09-17T15:49:59.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.186+0000 7f6ff0ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.186+0000 7f6ff0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.186+0000 7f6ff0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.186+0000 7f6ff0ff9640 10 monclient(hunting): my global_id is 5678 2024-09-17T15:49:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.186+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd80065c0 con 0x7f6fd80090e0 2024-09-17T15:49:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1664966957 0 0) 0x7f6fd80065c0 con 0x7f6fd80090e0 2024-09-17T15:49:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fd8009e20 con 0x7f6fd80090e0 2024-09-17T15:49:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6fe4005460 con 0x7f6fd80090e0 2024-09-17T15:49:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:49:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff0ff9640 10 monclient: dump: 2024-09-17T15:49:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.212 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.213 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.213 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.213 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.213 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.213 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.213 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.213 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.213 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.213 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.213 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3898581605 0 0) 0x7f6fd8009e20 con 0x7f6fd80090e0 2024-09-17T15:49:59.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 --> v1:172.21.15.64:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6fd800c3f0 con 0x7f6fd80090e0 2024-09-17T15:49:59.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 2183905317) 0x7f6fe4005610 con 0x7f6fd80090e0 2024-09-17T15:49:59.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:59.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:59.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 >> v1:172.21.15.64:6793/0 conn(0x7f6fd80090e0 legacy=0x7f6fd8009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff0ff9640 1 -- 172.21.15.64:0/1095682377 >> v1:172.21.15.64:6793/0 conn(0x7f6fd80090e0 legacy=0x7f6fd8009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:59.214 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:49:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:49:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:49:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:49:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:49:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 255834, 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:49:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:49:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:49:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:49:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:49:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:49:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:49:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:49:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.234 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:49:59.234 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:49:59.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff9a01640 1 -- 172.21.15.64:0/1095682377 >> v1:172.21.15.100:6812/4210963484 conn(0x7f6fd4040da0 legacy=0x7f6fd4043260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff9a01640 10 monclient: shutdown 2024-09-17T15:49:59.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff9a01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:59.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff9a01640 1 -- 172.21.15.64:0/1095682377 >> v1:172.21.15.64:6793/0 conn(0x7f6ff4135f40 legacy=0x7f6ff40a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff37fe640 1 -- 172.21.15.64:0/1095682377 reap_dead start 2024-09-17T15:49:59.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff37fe640 1 -- 172.21.15.64:0/1095682377 reap_dead start 2024-09-17T15:49:59.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff9a01640 1 -- 172.21.15.64:0/1095682377 shutdown_connections 2024-09-17T15:49:59.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.187+0000 7f6ff9a01640 1 -- 172.21.15.64:0/1095682377 >> 172.21.15.64:0/1095682377 conn(0x7f6ff406d9a0 msgr2=0x7f6ff413eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:59.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.188+0000 7f6ff9a01640 1 -- 172.21.15.64:0/1095682377 shutdown_connections 2024-09-17T15:49:59.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.188+0000 7f6ff9a01640 1 -- 172.21.15.64:0/1095682377 wait complete. 2024-09-17T15:49:59.236 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:49:59.237 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-17T15:49:59.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-17T15:49:59.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.337+0000 7f29e5552640 10 monclient: get_monmap_and_config 2024-09-17T15:49:59.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.337+0000 7f29e5552640 10 monclient: build_initial_monmap 2024-09-17T15:49:59.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.337+0000 7f29e5552640 10 monclient: monmap: 2024-09-17T15:49:59.339 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:59.339 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.339 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:49:59.338339+0000 2024-09-17T15:49:59.339 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:49:59.338339+0000 2024-09-17T15:49:59.339 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:59.339 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.339 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:59.340 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:59.340 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:59.340 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:59.340 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:59.340 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:59.340 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:59.340 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:59.340 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:59.340 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29e5552640 1 Processor -- start 2024-09-17T15:49:59.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29e5552640 1 -- start start 2024-09-17T15:49:59.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29e5552640 10 monclient: init 2024-09-17T15:49:59.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29e5552640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:59.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29e5552640 10 monclient: _add_conns ranks=[4,5,8,0,7,3,2,6,1] 2024-09-17T15:49:59.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29e5552640 10 monclient(hunting): picked mon.noname-c con 0x7f29e00a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:49:59.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29e5552640 10 monclient(hunting): picked mon.noname-h con 0x7f29e0135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:49:59.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29e5552640 10 monclient(hunting): picked mon.noname-e con 0x7f29e0135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:49:59.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29e5552640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29e009d6f0 con 0x7f29e00a5520 2024-09-17T15:49:59.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29e5552640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29e009d9d0 con 0x7f29e0135b50 2024-09-17T15:49:59.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29deffd640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f29e00a5520 0x7f29e00a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:52740/0 (socket says 172.21.15.64:52740) 2024-09-17T15:49:59.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29deffd640 1 -- 172.21.15.64:0/3153289590 learned_addr learned my addr 172.21.15.64:0/3153289590 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:59.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29e5552640 1 -- 172.21.15.64:0/3153289590 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29e009dcb0 con 0x7f29e0135f70 2024-09-17T15:49:59.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29e5552640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:59.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.338+0000 7f29e5552640 10 monclient(hunting): authenticate will time out at 997.013936s 2024-09-17T15:49:59.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 -- 172.21.15.64:0/3153289590 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3819887261 0 0) 0x7f29e009dcb0 con 0x7f29e0135f70 2024-09-17T15:49:59.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient(hunting): my global_id is 5687 2024-09-17T15:49:59.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 -- 172.21.15.64:0/3153289590 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29c8003520 con 0x7f29e0135f70 2024-09-17T15:49:59.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 -- 172.21.15.64:0/3153289590 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2831570879 0 0) 0x7f29e009d6f0 con 0x7f29e00a5520 2024-09-17T15:49:59.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient(hunting): my global_id is 5773 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 -- 172.21.15.64:0/3153289590 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29e009dcb0 con 0x7f29e00a5520 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 -- 172.21.15.64:0/3153289590 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1835448965 0 0) 0x7f29e009d9d0 con 0x7f29e0135b50 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient(hunting): my global_id is 124178 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 -- 172.21.15.64:0/3153289590 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29e009d6f0 con 0x7f29e0135b50 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 -- 172.21.15.64:0/3153289590 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 201539797 0 0) 0x7f29c8003520 con 0x7f29e0135f70 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 -- 172.21.15.64:0/3153289590 >> v1:172.21.15.100:6791/0 conn(0x7f29e0135b50 legacy=0x7f29e013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 -- 172.21.15.64:0/3153289590 >> v1:172.21.15.64:6791/0 conn(0x7f29e00a5520 legacy=0x7f29e00a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient: _finish_hunting 0 2024-09-17T15:49:59.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 monclient: found mon.noname-e 2024-09-17T15:49:59.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:59.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 -- 172.21.15.64:0/3153289590 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e009ccd0 con 0x7f29e0135f70 2024-09-17T15:49:59.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient: _finish_auth 0 2024-09-17T15:49:59.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient: _check_auth_tickets 2024-09-17T15:49:59.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:59.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 -- 172.21.15.64:0/3153289590 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29d8001bb0 con 0x7f29e0135f70 2024-09-17T15:49:59.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29e5552640 5 monclient: authenticate success, global_id 5687 2024-09-17T15:49:59.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29e5552640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:59.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:49:59.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient: dump: 2024-09-17T15:49:59.353 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.353 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.353 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.353 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.353 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.353 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.353 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.353 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.354 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.354 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.354 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.354 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.354 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.354 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.354 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.354 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29e5552640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:59.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 -- 172.21.15.64:0/3153289590 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f29d8001dd0 con 0x7f29e0135f70 2024-09-17T15:49:59.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:59.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 1 -- 172.21.15.64:0/3153289590 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29d80020c0 con 0x7f29e0135f70 2024-09-17T15:49:59.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29e5552640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:59.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29e5552640 4 set_mon_vals no callback set 2024-09-17T15:49:59.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:49:59.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29ddffb640 10 monclient: dump: 2024-09-17T15:49:59.355 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.355 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.356 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.356 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.356 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.356 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.356 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.356 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.356 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.356 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.356 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.356 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.356 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.356 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.356 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.357 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29e5552640 10 monclient: shutdown 2024-09-17T15:49:59.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29e5552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:59.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.339+0000 7f29e5552640 1 -- 172.21.15.64:0/3153289590 >> v1:172.21.15.64:6793/0 conn(0x7f29e0135f70 legacy=0x7f29e01415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.340+0000 7f29e5552640 1 -- 172.21.15.64:0/3153289590 shutdown_connections 2024-09-17T15:49:59.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.340+0000 7f29e5552640 1 -- 172.21.15.64:0/3153289590 >> 172.21.15.64:0/3153289590 conn(0x7f29e006db30 msgr2=0x7f29e00ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:59.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.340+0000 7f29e5552640 1 -- 172.21.15.64:0/3153289590 shutdown_connections 2024-09-17T15:49:59.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.340+0000 7f29e5552640 1 -- 172.21.15.64:0/3153289590 wait complete. 2024-09-17T15:49:59.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.340+0000 7f29e5552640 10 monclient: build_initial_monmap 2024-09-17T15:49:59.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.340+0000 7f29e5552640 10 monclient: monmap: 2024-09-17T15:49:59.357 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:59.357 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:59.357 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:49:59.358 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:49:59.358 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:59.358 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:49:59.358 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:59.358 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:59.358 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:59.358 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:59.358 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:59.358 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:59.358 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:59.358 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:59.358 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:59.358 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29e5552640 1 Processor -- start 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29e5552640 1 -- start start 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29e5552640 10 monclient: init 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29e5552640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29e5552640 10 monclient: _add_conns ranks=[6,8,7,5,0,1,3,4,2] 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29e5552640 10 monclient(hunting): picked mon.noname-d con 0x7f29e00a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29e5552640 10 monclient(hunting): picked mon.noname-e con 0x7f29e0135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29e5552640 10 monclient(hunting): picked mon.noname-i con 0x7f29e0135f70 addr v1:172.21.15.100:6792/0 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29e5552640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29e00a0970 con 0x7f29e00a5520 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29e5552640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29e0132970 con 0x7f29e0135f70 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29e5552640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29e01352c0 con 0x7f29e0135b50 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29e5552640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29e5552640 10 monclient(hunting): authenticate will time out at 997.016739s 2024-09-17T15:49:59.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29de7fc640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f29e0135b50 0x7f29e0132260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:58068/0 (socket says 172.21.15.64:58068) 2024-09-17T15:49:59.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29deffd640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f29e00a5520 0x7f29e0131b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:44026/0 (socket says 172.21.15.64:44026) 2024-09-17T15:49:59.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29de7fc640 1 -- 172.21.15.64:0/3299628103 learned_addr learned my addr 172.21.15.64:0/3299628103 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:59.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 435826736 0 0) 0x7f29e01352c0 con 0x7f29e0135b50 2024-09-17T15:49:59.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29c77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.341+0000 7f29c77fe640 10 monclient(hunting): my global_id is 5696 2024-09-17T15:49:59.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29cc003520 con 0x7f29e0135b50 2024-09-17T15:49:59.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3039468718 0 0) 0x7f29e00a0970 con 0x7f29e00a5520 2024-09-17T15:49:59.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): my global_id is 124152 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29e01352c0 con 0x7f29e00a5520 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4134832142 0 0) 0x7f29e0132970 con 0x7f29e0135f70 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): my global_id is 5560 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29e00a0970 con 0x7f29e0135f70 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3924766086 0 0) 0x7f29cc003520 con 0x7f29e0135b50 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29e0132970 con 0x7f29e0135b50 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1775696855 0 0) 0x7f29e01352c0 con 0x7f29e00a5520 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29cc003520 con 0x7f29e00a5520 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29d4001bb0 con 0x7f29e0135b50 2024-09-17T15:49:59.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:49:59.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): dump: 2024-09-17T15:49:59.362 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.362 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.362 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.362 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.362 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.362 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.362 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.362 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.362 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.363 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.363 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.363 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.363 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.363 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.363 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.363 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:49:59.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29d0001430 con 0x7f29e00a5520 2024-09-17T15:49:59.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:49:59.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): dump: 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.364 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1441700513 0 0) 0x7f29e00a0970 con 0x7f29e0135f70 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29e01352c0 con 0x7f29e0135f70 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29d8001410 con 0x7f29e0135f70 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient(hunting): dump: 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.365 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3928326545 0 0) 0x7f29e0132970 con 0x7f29e0135b50 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 >> v1:172.21.15.100:6792/0 conn(0x7f29e0135f70 legacy=0x7f29e01d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 >> v1:172.21.15.64:6792/0 conn(0x7f29e00a5520 legacy=0x7f29e0131b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 monclient: found mon.e 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e0134f20 con 0x7f29e0135b50 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient: _finish_auth 0 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29e5552640 5 monclient: authenticate success, global_id 5696 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29e5552640 10 monclient: _renew_subs 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29e5552640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29e5552640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29e009fb00 con 0x7f29e0135b50 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29e5552640 10 monclient: _renew_subs 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29e5552640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29e5552640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f29e00a0690 con 0x7f29e0135b50 2024-09-17T15:49:59.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f29d40038c0 con 0x7f29e0135b50 2024-09-17T15:49:59.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:59.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29d4003e30 con 0x7f29e0135b50 2024-09-17T15:49:59.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29dd7fa640 4 set_mon_vals no callback set 2024-09-17T15:49:59.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:49:59.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.342+0000 7f29c77fe640 10 monclient: dump: 2024-09-17T15:49:59.368 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.368 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.368 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.368 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.368 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.368 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.343+0000 7f29e5552640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.343+0000 7f29e5552640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f29c00065c0 con 0x7f29c0002860 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.343+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f29d4004050 con 0x7f29e0135b50 2024-09-17T15:49:59.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.343+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(193..193 src has 1..193) ==== 3506+0+0 (unknown 1845487835 0 0) 0x7f29d404ebb0 con 0x7f29e0135b50 2024-09-17T15:49:59.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3876465549 0 0) 0x7f29c00065c0 con 0x7f29c0002860 2024-09-17T15:49:59.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 10 monclient(hunting): my global_id is 5842 2024-09-17T15:49:59.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29e0132970 con 0x7f29c0002860 2024-09-17T15:49:59.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29df7fe640 10 monclient: get_auth_request con 0x7f29cc040bd0 auth_method 0 2024-09-17T15:49:59.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3292223545 0 0) 0x7f29e0132970 con 0x7f29c0002860 2024-09-17T15:49:59.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29c00065c0 con 0x7f29c0002860 2024-09-17T15:49:59.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29d0004dd0 con 0x7f29c0002860 2024-09-17T15:49:59.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:49:59.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 10 monclient: dump: 2024-09-17T15:49:59.371 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.371 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.371 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.371 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.371 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.371 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.371 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.371 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.371 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.371 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.371 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.371 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.371 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.372 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.372 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.372 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2071566667 0 0) 0x7f29c00065c0 con 0x7f29c0002860 2024-09-17T15:49:59.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.344+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.100:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29c000a160 con 0x7f29c0002860 2024-09-17T15:49:59.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.345+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f29d0007090 con 0x7f29c0002860 2024-09-17T15:49:59.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.345+0000 7f29c77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:59.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.345+0000 7f29c77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:59.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.345+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 >> v1:172.21.15.100:6789/0 conn(0x7f29c0002860 legacy=0x7f29c0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.345+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 >> v1:172.21.15.100:6789/0 conn(0x7f29c0002860 legacy=0x7f29c0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:59.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.351+0000 7f29e5552640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:59.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.351+0000 7f29e5552640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f29c0009e70 con 0x7f29c0009130 2024-09-17T15:49:59.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.351+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3468558897 0 0) 0x7f29c0009e70 con 0x7f29c0009130 2024-09-17T15:49:59.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.351+0000 7f29c77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.351+0000 7f29c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.351+0000 7f29c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.351+0000 7f29c77fe640 10 monclient(hunting): my global_id is 5851 2024-09-17T15:49:59.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.351+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29c00065c0 con 0x7f29c0009130 2024-09-17T15:49:59.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.352+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1796958602 0 0) 0x7f29c00065c0 con 0x7f29c0009130 2024-09-17T15:49:59.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.352+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29c0009e70 con 0x7f29c0009130 2024-09-17T15:49:59.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.352+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f29d0001850 con 0x7f29c0009130 2024-09-17T15:49:59.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.352+0000 7f29c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.352+0000 7f29c77fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:49:59.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.352+0000 7f29c77fe640 10 monclient: dump: 2024-09-17T15:49:59.375 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.375 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.375 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.375 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.375 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.375 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.375 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.375 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.375 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.375 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.376 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.376 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.376 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.376 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.376 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.376 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.352+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2871493496 0 0) 0x7f29c0009e70 con 0x7f29c0009130 2024-09-17T15:49:59.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.352+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 --> v1:172.21.15.100:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f29c000c410 con 0x7f29c0009130 2024-09-17T15:49:59.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.352+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7546 (unknown 0 0 3425917630) 0x7f29d000a000 con 0x7f29c0009130 2024-09-17T15:49:59.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.352+0000 7f29c77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:59.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.352+0000 7f29c77fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:59.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.352+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 >> v1:172.21.15.100:6789/0 conn(0x7f29c0009130 legacy=0x7f29c00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.352+0000 7f29c77fe640 1 -- 172.21.15.64:0/3299628103 >> v1:172.21.15.100:6789/0 conn(0x7f29c0009130 legacy=0x7f29c00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:59.377 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:49:59.377 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:49:59.377 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:49:59.377 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "leader", 2024-09-17T15:49:59.377 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:49:59.377 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 256003, 2024-09-17T15:49:59.377 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:49:59.377 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:49:59.377 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:49:59.377 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:49:59.377 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:49:59.378 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:49:59.378 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:49:59.378 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:49:59.378 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:49:59.378 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:49:59.378 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.378 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:49:59.378 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:59.378 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:59.378 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:49:59.378 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.378 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:49:59.380 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:49:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:49:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:49:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:49:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:49:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:49:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:49:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:59.381 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:49:59.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.383 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:49:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:49:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:49:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:49:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.384 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:49:59.385 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.385 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.385 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:49:59.385 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:49:59.385 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.385 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.385 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.385 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.386 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.388 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:49:59.388 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:49:59.388 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.388 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.388 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.388 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:49:59.388 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:49:59.389 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.389 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.389 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.389 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:49:59.389 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.389 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.390 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:49:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:49:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:49:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:49:59.394 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:49:59.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:49:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 4 2024-09-17T15:49:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.397 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:49:59.397 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:49:59.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.353+0000 7f29e5552640 1 -- 172.21.15.64:0/3299628103 >> v1:172.21.15.100:6812/4210963484 conn(0x7f29cc040bd0 legacy=0x7f29cc043090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.353+0000 7f29e5552640 10 monclient: shutdown 2024-09-17T15:49:59.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.353+0000 7f29e5552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:59.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.353+0000 7f29e5552640 1 -- 172.21.15.64:0/3299628103 >> v1:172.21.15.64:6793/0 conn(0x7f29e0135b50 legacy=0x7f29e0132260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.353+0000 7f29df7fe640 1 -- 172.21.15.64:0/3299628103 reap_dead start 2024-09-17T15:49:59.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.353+0000 7f29df7fe640 1 -- 172.21.15.64:0/3299628103 reap_dead start 2024-09-17T15:49:59.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.353+0000 7f29e5552640 1 -- 172.21.15.64:0/3299628103 shutdown_connections 2024-09-17T15:49:59.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.353+0000 7f29e5552640 1 -- 172.21.15.64:0/3299628103 >> 172.21.15.64:0/3299628103 conn(0x7f29e006db30 msgr2=0x7f29e012c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:59.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.353+0000 7f29e5552640 1 -- 172.21.15.64:0/3299628103 shutdown_connections 2024-09-17T15:49:59.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.353+0000 7f29e5552640 1 -- 172.21.15.64:0/3299628103 wait complete. 2024-09-17T15:49:59.399 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:49:59.399 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-17T15:49:59.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-17T15:49:59.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.507+0000 7f934d8be640 10 monclient: get_monmap_and_config 2024-09-17T15:49:59.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.507+0000 7f934d8be640 10 monclient: build_initial_monmap 2024-09-17T15:49:59.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.507+0000 7f934d8be640 10 monclient: monmap: 2024-09-17T15:49:59.509 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:59.509 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.509 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:49:59.508006+0000 2024-09-17T15:49:59.509 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:49:59.508006+0000 2024-09-17T15:49:59.509 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:59.509 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.509 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:59.509 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:59.509 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:59.509 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:59.510 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:59.510 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:59.510 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:59.510 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:59.510 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:59.510 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.507+0000 7f934d8be640 1 Processor -- start 2024-09-17T15:49:59.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.507+0000 7f934d8be640 1 -- start start 2024-09-17T15:49:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.507+0000 7f934d8be640 10 monclient: init 2024-09-17T15:49:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f934d8be640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f934d8be640 10 monclient: _add_conns ranks=[2,3,8,6,0,4,7,1,5] 2024-09-17T15:49:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f934d8be640 10 monclient(hunting): picked mon.noname-b con 0x7f93480a54b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:49:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f934d8be640 10 monclient(hunting): picked mon.noname-g con 0x7f9348135c80 addr v1:172.21.15.100:6790/0 2024-09-17T15:49:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f934d8be640 10 monclient(hunting): picked mon.noname-e con 0x7f93481360a0 addr v1:172.21.15.64:6793/0 2024-09-17T15:49:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f934d8be640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93480aacb0 con 0x7f93480a54b0 2024-09-17T15:49:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f934d8be640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f934809d780 con 0x7f9348135c80 2024-09-17T15:49:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f934d8be640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f934809da60 con 0x7f93481360a0 2024-09-17T15:49:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f934d8be640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f934d8be640 10 monclient(hunting): authenticate will time out at 997.183330s 2024-09-17T15:49:59.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f93477fe640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f93481360a0 0x7f9348141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:58080/0 (socket says 172.21.15.64:58080) 2024-09-17T15:49:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f93477fe640 1 -- 172.21.15.64:0/7912886 learned_addr learned my addr 172.21.15.64:0/7912886 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9346ffd640 1 --1- 172.21.15.64:0/7912886 >> v1:172.21.15.64:6790/0 conn(0x7f93480a54b0 0x7f93480a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46930/0 (socket says 172.21.15.64:46930) 2024-09-17T15:49:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 1 -- 172.21.15.64:0/7912886 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 698857050 0 0) 0x7f93480aacb0 con 0x7f93480a54b0 2024-09-17T15:49:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 10 monclient(hunting): my global_id is 124148 2024-09-17T15:49:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 1 -- 172.21.15.64:0/7912886 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9334003520 con 0x7f93480a54b0 2024-09-17T15:49:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 1 -- 172.21.15.64:0/7912886 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2002283264 0 0) 0x7f934809da60 con 0x7f93481360a0 2024-09-17T15:49:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 10 monclient(hunting): my global_id is 5705 2024-09-17T15:49:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 1 -- 172.21.15.64:0/7912886 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93480aacb0 con 0x7f93481360a0 2024-09-17T15:49:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 1 -- 172.21.15.64:0/7912886 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1145420738 0 0) 0x7f93480aacb0 con 0x7f93481360a0 2024-09-17T15:49:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 1 -- 172.21.15.64:0/7912886 >> v1:172.21.15.100:6790/0 conn(0x7f9348135c80 legacy=0x7f934813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 1 -- 172.21.15.64:0/7912886 >> v1:172.21.15.64:6790/0 conn(0x7f93480a54b0 legacy=0x7f93480a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:49:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 1 monclient: found mon.noname-e 2024-09-17T15:49:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 1 -- 172.21.15.64:0/7912886 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934809dd40 con 0x7f93481360a0 2024-09-17T15:49:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 10 monclient: _finish_auth 0 2024-09-17T15:49:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:49:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f934d8be640 5 monclient: authenticate success, global_id 5705 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f934d8be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.508+0000 7f9345ffb640 1 -- 172.21.15.64:0/7912886 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9340001b90 con 0x7f93481360a0 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f9345ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f9345ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f9345ffb640 10 monclient: dump: 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.521 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f934d8be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f9345ffb640 1 -- 172.21.15.64:0/7912886 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9340001db0 con 0x7f93481360a0 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f9345ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f9345ffb640 1 -- 172.21.15.64:0/7912886 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9340004f60 con 0x7f93481360a0 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f934d8be640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f934d8be640 4 set_mon_vals no callback set 2024-09-17T15:49:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f9345ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f9345ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:49:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f9345ffb640 10 monclient: dump: 2024-09-17T15:49:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.523 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.524 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f934d8be640 10 monclient: shutdown 2024-09-17T15:49:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f934d8be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f934d8be640 1 -- 172.21.15.64:0/7912886 >> v1:172.21.15.64:6793/0 conn(0x7f93481360a0 legacy=0x7f9348141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f934d8be640 1 -- 172.21.15.64:0/7912886 shutdown_connections 2024-09-17T15:49:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f934d8be640 1 -- 172.21.15.64:0/7912886 >> 172.21.15.64:0/7912886 conn(0x7f934806daa0 msgr2=0x7f93480ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f934d8be640 1 -- 172.21.15.64:0/7912886 shutdown_connections 2024-09-17T15:49:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.509+0000 7f934d8be640 1 -- 172.21.15.64:0/7912886 wait complete. 2024-09-17T15:49:59.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.510+0000 7f934d8be640 10 monclient: build_initial_monmap 2024-09-17T15:49:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.510+0000 7f934d8be640 10 monclient: monmap: 2024-09-17T15:49:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:49:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:49:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:49:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:59.525 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.510+0000 7f934d8be640 1 Processor -- start 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.510+0000 7f934d8be640 1 -- start start 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.510+0000 7f934d8be640 10 monclient: init 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f934d8be640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f934d8be640 10 monclient: _add_conns ranks=[8,6,7,0,2,3,1,4,5] 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f934d8be640 10 monclient(hunting): picked mon.noname-e con 0x7f93480a54b0 addr v1:172.21.15.64:6793/0 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f934d8be640 10 monclient(hunting): picked mon.noname-d con 0x7f9348135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f934d8be640 10 monclient(hunting): picked mon.noname-i con 0x7f93481360a0 addr v1:172.21.15.100:6792/0 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f934d8be640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f934809e550 con 0x7f9348135c80 2024-09-17T15:49:59.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f934d8be640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93480a08b0 con 0x7f93481360a0 2024-09-17T15:49:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f934d8be640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93480a0010 con 0x7f93480a54b0 2024-09-17T15:49:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f934d8be640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f934d8be640 10 monclient(hunting): authenticate will time out at 997.186296s 2024-09-17T15:49:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f9346ffd640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f93480a54b0 0x7f9348131980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:58090/0 (socket says 172.21.15.64:58090) 2024-09-17T15:49:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f9346ffd640 1 -- 172.21.15.64:0/519456183 learned_addr learned my addr 172.21.15.64:0/519456183 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3801614041 0 0) 0x7f93480a0010 con 0x7f93480a54b0 2024-09-17T15:49:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): my global_id is 5714 2024-09-17T15:49:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9328003520 con 0x7f93480a54b0 2024-09-17T15:49:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2927802719 0 0) 0x7f934809e550 con 0x7f9348135c80 2024-09-17T15:49:59.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): my global_id is 124161 2024-09-17T15:49:59.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93480a0010 con 0x7f9348135c80 2024-09-17T15:49:59.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 590883035 0 0) 0x7f93480a08b0 con 0x7f93481360a0 2024-09-17T15:49:59.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): my global_id is 5569 2024-09-17T15:49:59.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f934809e550 con 0x7f93481360a0 2024-09-17T15:49:59.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 98329808 0 0) 0x7f9328003520 con 0x7f93480a54b0 2024-09-17T15:49:59.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93480a08b0 con 0x7f93480a54b0 2024-09-17T15:49:59.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9338001630 con 0x7f93480a54b0 2024-09-17T15:49:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:49:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): dump: 2024-09-17T15:49:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.529 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 579334266 0 0) 0x7f93480a0010 con 0x7f9348135c80 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9328003520 con 0x7f9348135c80 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9330001a90 con 0x7f9348135c80 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.511+0000 7f933ffff640 10 monclient(hunting): dump: 2024-09-17T15:49:59.530 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.531 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.532 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4142899595 0 0) 0x7f93480a08b0 con 0x7f93480a54b0 2024-09-17T15:49:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 >> v1:172.21.15.100:6792/0 conn(0x7f93481360a0 legacy=0x7f93481327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 >> v1:172.21.15.64:6792/0 conn(0x7f9348135c80 legacy=0x7f9348132090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:49:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 1 monclient: found mon.e 2024-09-17T15:49:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93480a4ed0 con 0x7f93480a54b0 2024-09-17T15:49:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 10 monclient: _finish_auth 0 2024-09-17T15:49:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:49:59.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f934d8be640 5 monclient: authenticate success, global_id 5714 2024-09-17T15:49:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f934d8be640 10 monclient: _renew_subs 2024-09-17T15:49:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f934d8be640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f934d8be640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f93480a5180 con 0x7f93480a54b0 2024-09-17T15:49:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f934d8be640 10 monclient: _renew_subs 2024-09-17T15:49:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f934d8be640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f934d8be640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f93481d7010 con 0x7f93480a54b0 2024-09-17T15:49:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9338004a90 con 0x7f93480a54b0 2024-09-17T15:49:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9338004f80 con 0x7f93480a54b0 2024-09-17T15:49:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f93457fa640 4 set_mon_vals no callback set 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 10 monclient: dump: 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.512+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f9338011860 con 0x7f93480a54b0 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f934d8be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f934d8be640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f932c0065c0 con 0x7f932c002860 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(194..194 src has 1..194) ==== 3898+0+0 (unknown 3438653403 0 0) 0x7f933804ec50 con 0x7f93480a54b0 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f93477fe640 10 monclient: get_auth_request con 0x7f9328040a20 auth_method 0 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 626035300 0 0) 0x7f932c0065c0 con 0x7f932c002860 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f933ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f933ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f933ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f933ffff640 10 monclient(hunting): my global_id is 114807 2024-09-17T15:49:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93480a08b0 con 0x7f932c002860 2024-09-17T15:49:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3457012585 0 0) 0x7f93480a08b0 con 0x7f932c002860 2024-09-17T15:49:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f932c0065c0 con 0x7f932c002860 2024-09-17T15:49:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9330003a70 con 0x7f932c002860 2024-09-17T15:49:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f933ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f933ffff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:49:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.513+0000 7f933ffff640 10 monclient: dump: 2024-09-17T15:49:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.538 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.514+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 555436341 0 0) 0x7f932c0065c0 con 0x7f932c002860 2024-09-17T15:49:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.514+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.100:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f932c00a160 con 0x7f932c002860 2024-09-17T15:49:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.514+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9330003d60 con 0x7f932c002860 2024-09-17T15:49:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.514+0000 7f933ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.514+0000 7f933ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.514+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 >> v1:172.21.15.100:6790/0 conn(0x7f932c002860 legacy=0x7f932c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.514+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 >> v1:172.21.15.100:6790/0 conn(0x7f932c002860 legacy=0x7f932c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.520+0000 7f934d8be640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.520+0000 7f934d8be640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f932c009e50 con 0x7f932c009110 2024-09-17T15:49:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.520+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1607650674 0 0) 0x7f932c009e50 con 0x7f932c009110 2024-09-17T15:49:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.520+0000 7f933ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.520+0000 7f933ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.520+0000 7f933ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.520+0000 7f933ffff640 10 monclient(hunting): my global_id is 114816 2024-09-17T15:49:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.520+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f932c0065c0 con 0x7f932c009110 2024-09-17T15:49:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.521+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 402181876 0 0) 0x7f932c0065c0 con 0x7f932c009110 2024-09-17T15:49:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.521+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f932c009e50 con 0x7f932c009110 2024-09-17T15:49:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.521+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f93300044d0 con 0x7f932c009110 2024-09-17T15:49:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.521+0000 7f933ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.521+0000 7f933ffff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:49:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.521+0000 7f933ffff640 10 monclient: dump: 2024-09-17T15:49:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.521+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4044516057 0 0) 0x7f932c009e50 con 0x7f932c009110 2024-09-17T15:49:59.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.521+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 --> v1:172.21.15.100:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f932c00c3a0 con 0x7f932c009110 2024-09-17T15:49:59.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.521+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 4186695804) 0x7f9330006770 con 0x7f932c009110 2024-09-17T15:49:59.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.521+0000 7f933ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:59.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.521+0000 7f933ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:59.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.521+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 >> v1:172.21.15.100:6790/0 conn(0x7f932c009110 legacy=0x7f932c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.521+0000 7f933ffff640 1 -- 172.21.15.64:0/519456183 >> v1:172.21.15.100:6790/0 conn(0x7f932c009110 legacy=0x7f932c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:59.543 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:49:59.543 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:49:59.543 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:49:59.543 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:49:59.543 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:49:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 122744, 2024-09-17T15:49:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:49:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:49:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:49:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:49:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:49:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:49:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:49:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:49:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:49:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:49:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.545 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.546 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.546 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:59.546 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:49:59.546 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.546 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.546 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.546 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.546 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.546 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.546 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.546 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.546 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.546 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.547 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.547 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.547 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:49:59.547 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:49:59.547 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:49:59.547 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:49:59.547 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:49:59.547 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:49:59.547 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:59.547 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:59.547 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:49:59.547 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:59.548 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:49:59.548 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:59.548 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:49:59.548 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:59.548 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:49:59.548 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:59.548 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:49:59.548 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.548 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.548 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.548 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.548 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.548 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.549 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.551 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:49:59.551 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:49:59.551 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.551 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.551 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.551 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:49:59.551 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.551 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.551 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:49:59.551 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.553 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.553 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:49:59.553 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:49:59.553 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.553 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.553 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.553 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:49:59.553 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:49:59.554 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.554 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.554 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.554 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:49:59.554 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.554 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.554 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:49:59.554 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:49:59.554 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.554 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.555 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:49:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:49:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:49:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:49:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.557 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:49:59.559 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.561 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.562 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:59.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.562 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.562 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:49:59.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.562 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.562 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.562 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:59.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.562 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:49:59.562 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:49:59.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.522+0000 7f934d8be640 1 -- 172.21.15.64:0/519456183 >> v1:172.21.15.100:6812/4210963484 conn(0x7f9328040a20 legacy=0x7f9328042ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.522+0000 7f934d8be640 10 monclient: shutdown 2024-09-17T15:49:59.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.522+0000 7f934d8be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:59.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.522+0000 7f934d8be640 1 -- 172.21.15.64:0/519456183 >> v1:172.21.15.64:6793/0 conn(0x7f93480a54b0 legacy=0x7f9348131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.522+0000 7f93477fe640 1 -- 172.21.15.64:0/519456183 reap_dead start 2024-09-17T15:49:59.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.522+0000 7f93477fe640 1 -- 172.21.15.64:0/519456183 reap_dead start 2024-09-17T15:49:59.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.522+0000 7f934d8be640 1 -- 172.21.15.64:0/519456183 shutdown_connections 2024-09-17T15:49:59.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.522+0000 7f934d8be640 1 -- 172.21.15.64:0/519456183 >> 172.21.15.64:0/519456183 conn(0x7f934806daa0 msgr2=0x7f93480a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:59.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.522+0000 7f934d8be640 1 -- 172.21.15.64:0/519456183 shutdown_connections 2024-09-17T15:49:59.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.522+0000 7f934d8be640 1 -- 172.21.15.64:0/519456183 wait complete. 2024-09-17T15:49:59.565 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:49:59.565 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-17T15:49:59.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-17T15:49:59.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.665+0000 7ff394905640 10 monclient: get_monmap_and_config 2024-09-17T15:49:59.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.665+0000 7ff394905640 10 monclient: build_initial_monmap 2024-09-17T15:49:59.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.665+0000 7ff394905640 10 monclient: monmap: 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:49:59.666719+0000 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:49:59.666719+0000 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:59.668 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:59.669 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:59.669 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.666+0000 7ff394905640 1 Processor -- start 2024-09-17T15:49:59.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.666+0000 7ff394905640 1 -- start start 2024-09-17T15:49:59.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.666+0000 7ff394905640 10 monclient: init 2024-09-17T15:49:59.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff394905640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:59.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff394905640 10 monclient: _add_conns ranks=[2,8,5,1,6,0,3,4,7] 2024-09-17T15:49:59.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff394905640 10 monclient(hunting): picked mon.noname-b con 0x7ff3900a54b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:49:59.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff394905640 10 monclient(hunting): picked mon.noname-e con 0x7ff390135c80 addr v1:172.21.15.64:6793/0 2024-09-17T15:49:59.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff394905640 10 monclient(hunting): picked mon.noname-h con 0x7ff3901360a0 addr v1:172.21.15.100:6791/0 2024-09-17T15:49:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff394905640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3900aacb0 con 0x7ff3900a54b0 2024-09-17T15:49:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff394905640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff39009d780 con 0x7ff3901360a0 2024-09-17T15:49:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff394905640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff39009da60 con 0x7ff390135c80 2024-09-17T15:49:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff394905640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff394905640 10 monclient(hunting): authenticate will time out at 997.342208s 2024-09-17T15:49:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38dd9b640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7ff390135c80 0x7ff39013df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:58102/0 (socket says 172.21.15.64:58102) 2024-09-17T15:49:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38e59c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7ff3900a54b0 0x7ff3900a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46944/0 (socket says 172.21.15.64:46944) 2024-09-17T15:49:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38dd9b640 1 -- 172.21.15.64:0/1158777985 learned_addr learned my addr 172.21.15.64:0/1158777985 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 1 -- 172.21.15.64:0/1158777985 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1268224028 0 0) 0x7ff3900aacb0 con 0x7ff3900a54b0 2024-09-17T15:49:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient(hunting): my global_id is 124157 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 1 -- 172.21.15.64:0/1158777985 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff37c003520 con 0x7ff3900a54b0 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 1 -- 172.21.15.64:0/1158777985 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 754465331 0 0) 0x7ff39009da60 con 0x7ff390135c80 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient(hunting): my global_id is 5723 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 1 -- 172.21.15.64:0/1158777985 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3900aacb0 con 0x7ff390135c80 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 1 -- 172.21.15.64:0/1158777985 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 817446079 0 0) 0x7ff37c003520 con 0x7ff3900a54b0 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 1 -- 172.21.15.64:0/1158777985 >> v1:172.21.15.64:6793/0 conn(0x7ff390135c80 legacy=0x7ff39013df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 1 -- 172.21.15.64:0/1158777985 >> v1:172.21.15.100:6791/0 conn(0x7ff3901360a0 legacy=0x7ff390141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient: _finish_hunting 0 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 1 monclient: found mon.noname-b 2024-09-17T15:49:59.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:49:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 1 -- 172.21.15.64:0/1158777985 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39009dd40 con 0x7ff3900a54b0 2024-09-17T15:49:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient: _finish_auth 0 2024-09-17T15:49:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:49:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff394905640 5 monclient: authenticate success, global_id 124157 2024-09-17T15:49:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 1 -- 172.21.15.64:0/1158777985 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff378001830 con 0x7ff3900a54b0 2024-09-17T15:49:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff38d59a640 10 monclient: dump: 2024-09-17T15:49:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.679 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.667+0000 7ff394905640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:59.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff38d59a640 1 -- 172.21.15.64:0/1158777985 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff3780019d0 con 0x7ff3900a54b0 2024-09-17T15:49:59.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff38d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:59.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff394905640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:59.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff38d59a640 1 -- 172.21.15.64:0/1158777985 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff378004b20 con 0x7ff3900a54b0 2024-09-17T15:49:59.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff38d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff394905640 4 set_mon_vals no callback set 2024-09-17T15:49:59.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff38d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:59.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff38d59a640 10 monclient: dump: 2024-09-17T15:49:59.681 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.681 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.681 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.681 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.681 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff394905640 10 monclient: shutdown 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff394905640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:59.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff394905640 1 -- 172.21.15.64:0/1158777985 >> v1:172.21.15.64:6790/0 conn(0x7ff3900a54b0 legacy=0x7ff3900a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff394905640 1 -- 172.21.15.64:0/1158777985 shutdown_connections 2024-09-17T15:49:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff394905640 1 -- 172.21.15.64:0/1158777985 >> 172.21.15.64:0/1158777985 conn(0x7ff39006daa0 msgr2=0x7ff3900ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff394905640 1 -- 172.21.15.64:0/1158777985 shutdown_connections 2024-09-17T15:49:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.668+0000 7ff394905640 1 -- 172.21.15.64:0/1158777985 wait complete. 2024-09-17T15:49:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.669+0000 7ff394905640 10 monclient: build_initial_monmap 2024-09-17T15:49:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.669+0000 7ff394905640 10 monclient: monmap: 2024-09-17T15:49:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:49:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:49:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:49:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:59.684 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:59.684 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:59.684 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:59.684 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:59.684 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:59.684 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:59.684 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:59.684 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:59.684 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.669+0000 7ff394905640 1 Processor -- start 2024-09-17T15:49:59.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.669+0000 7ff394905640 1 -- start start 2024-09-17T15:49:59.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.669+0000 7ff394905640 10 monclient: init 2024-09-17T15:49:59.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:59.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 10 monclient: _add_conns ranks=[4,5,3,1,8,6,0,7,2] 2024-09-17T15:49:59.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 10 monclient(hunting): picked mon.noname-c con 0x7ff3900a54b0 addr v1:172.21.15.64:6791/0 2024-09-17T15:49:59.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 10 monclient(hunting): picked mon.noname-h con 0x7ff390135c80 addr v1:172.21.15.100:6791/0 2024-09-17T15:49:59.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 10 monclient(hunting): picked mon.noname-g con 0x7ff3901360a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:49:59.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3900a03f0 con 0x7ff3901360a0 2024-09-17T15:49:59.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3901350f0 con 0x7ff3900a54b0 2024-09-17T15:49:59.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3900a2470 con 0x7ff390135c80 2024-09-17T15:49:59.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:59.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 10 monclient(hunting): authenticate will time out at 997.345191s 2024-09-17T15:49:59.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff38e59c640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7ff3900a54b0 0x7ff39009ed30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:52748/0 (socket says 172.21.15.64:52748) 2024-09-17T15:49:59.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff38e59c640 1 -- 172.21.15.64:0/2474723998 learned_addr learned my addr 172.21.15.64:0/2474723998 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:59.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 882362895 0 0) 0x7ff3901350f0 con 0x7ff3900a54b0 2024-09-17T15:49:59.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): my global_id is 5782 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff370003520 con 0x7ff3900a54b0 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 418304843 0 0) 0x7ff3900a03f0 con 0x7ff3901360a0 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): my global_id is 114825 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3901350f0 con 0x7ff3901360a0 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1318215274 0 0) 0x7ff3900a2470 con 0x7ff390135c80 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): my global_id is 124196 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3900a03f0 con 0x7ff390135c80 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3734647085 0 0) 0x7ff370003520 con 0x7ff3900a54b0 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3900a2470 con 0x7ff3900a54b0 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff3780047e0 con 0x7ff3900a54b0 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): dump: 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.688 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.688 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.688 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.688 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.688 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.688 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.688 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.688 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.688 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.688 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:49:59.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1071904732 0 0) 0x7ff3900a2470 con 0x7ff3900a54b0 2024-09-17T15:49:59.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 >> v1:172.21.15.100:6791/0 conn(0x7ff390135c80 legacy=0x7ff39009f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 >> v1:172.21.15.100:6790/0 conn(0x7ff3901360a0 legacy=0x7ff3901d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:49:59.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 monclient: found mon.c 2024-09-17T15:49:59.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:59.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:49:59.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3900a2750 con 0x7ff3900a54b0 2024-09-17T15:49:59.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient: _finish_auth 0 2024-09-17T15:49:59.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:49:59.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff386ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:59.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 5 monclient: authenticate success, global_id 5782 2024-09-17T15:49:59.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 10 monclient: _renew_subs 2024-09-17T15:49:59.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:49:59.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.670+0000 7ff394905640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff3900a0f50 con 0x7ff3900a54b0 2024-09-17T15:49:59.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.671+0000 7ff394905640 10 monclient: _renew_subs 2024-09-17T15:49:59.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.671+0000 7ff394905640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:49:59.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.671+0000 7ff394905640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7ff39009fd30 con 0x7ff3900a54b0 2024-09-17T15:49:59.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.671+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff3780011b0 con 0x7ff3900a54b0 2024-09-17T15:49:59.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.671+0000 7ff386ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:59.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.671+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff3780019d0 con 0x7ff3900a54b0 2024-09-17T15:49:59.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.671+0000 7ff386ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.671+0000 7ff38cd99640 4 set_mon_vals no callback set 2024-09-17T15:49:59.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.671+0000 7ff386ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:59.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.671+0000 7ff386ffd640 10 monclient: dump: 2024-09-17T15:49:59.691 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.691 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.691 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.691 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.691 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.691 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.671+0000 7ff394905640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.671+0000 7ff394905640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff3740065c0 con 0x7ff374002860 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.671+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7ff378011860 con 0x7ff3900a54b0 2024-09-17T15:49:59.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(194..194 src has 1..194) ==== 3898+0+0 (unknown 3438653403 0 0) 0x7ff37804ec50 con 0x7ff3900a54b0 2024-09-17T15:49:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2140282407 0 0) 0x7ff3740065c0 con 0x7ff374002860 2024-09-17T15:49:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 10 monclient(hunting): my global_id is 124205 2024-09-17T15:49:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3900a2470 con 0x7ff374002860 2024-09-17T15:49:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff38ed9d640 10 monclient: get_auth_request con 0x7ff370040740 auth_method 0 2024-09-17T15:49:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 127403815 0 0) 0x7ff3900a2470 con 0x7ff374002860 2024-09-17T15:49:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3740065c0 con 0x7ff374002860 2024-09-17T15:49:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff3800012e0 con 0x7ff374002860 2024-09-17T15:49:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:49:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 10 monclient: dump: 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.694 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.695 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.695 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 873039662 0 0) 0x7ff3740065c0 con 0x7ff374002860 2024-09-17T15:49:59.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.672+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.100:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff37400a160 con 0x7ff374002860 2024-09-17T15:49:59.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.673+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff380001500 con 0x7ff374002860 2024-09-17T15:49:59.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.673+0000 7ff386ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:59.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.673+0000 7ff386ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:59.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.673+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 >> v1:172.21.15.100:6791/0 conn(0x7ff374002860 legacy=0x7ff374004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.673+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 >> v1:172.21.15.100:6791/0 conn(0x7ff374002860 legacy=0x7ff374004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:59.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.679+0000 7ff394905640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:59.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.679+0000 7ff394905640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff374009e70 con 0x7ff374009130 2024-09-17T15:49:59.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.679+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026353816 0 0) 0x7ff374009e70 con 0x7ff374009130 2024-09-17T15:49:59.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.679+0000 7ff386ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.679+0000 7ff386ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.679+0000 7ff386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.679+0000 7ff386ffd640 10 monclient(hunting): my global_id is 124214 2024-09-17T15:49:59.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.679+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3740065c0 con 0x7ff374009130 2024-09-17T15:49:59.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3033515088 0 0) 0x7ff3740065c0 con 0x7ff374009130 2024-09-17T15:49:59.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff374009e70 con 0x7ff374009130 2024-09-17T15:49:59.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ff380004c20 con 0x7ff374009130 2024-09-17T15:49:59.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff386ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff386ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:49:59.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff386ffd640 10 monclient: dump: 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.698 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.699 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.699 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1688825950 0 0) 0x7ff374009e70 con 0x7ff374009130 2024-09-17T15:49:59.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 --> v1:172.21.15.100:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff37400c410 con 0x7ff374009130 2024-09-17T15:49:59.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 2200722616) 0x7ff380006230 con 0x7ff374009130 2024-09-17T15:49:59.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff386ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:59.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff386ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:59.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 >> v1:172.21.15.100:6791/0 conn(0x7ff374009130 legacy=0x7ff3740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff386ffd640 1 -- 172.21.15.64:0/2474723998 >> v1:172.21.15.100:6791/0 conn(0x7ff374009130 legacy=0x7ff3740095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:59.699 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 6602, 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:49:59.700 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:49:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:49:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:49:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:49:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.701 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.702 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:59.703 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:49:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:49:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:49:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:49:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:49:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.704 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:49:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:49:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.705 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.706 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:49:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:49:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:49:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.707 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:49:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:49:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:49:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:49:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.709 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.710 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:49:59.712 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:49:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:49:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:49:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:49:59.713 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:49:59.715 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.716 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:49:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:49:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:49:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:49:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.717 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:49:59.718 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:49:59.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff394905640 1 -- 172.21.15.64:0/2474723998 >> v1:172.21.15.100:6812/4210963484 conn(0x7ff370040740 legacy=0x7ff370042c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff394905640 10 monclient: shutdown 2024-09-17T15:49:59.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff394905640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:59.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff394905640 1 -- 172.21.15.64:0/2474723998 >> v1:172.21.15.64:6791/0 conn(0x7ff3900a54b0 legacy=0x7ff39009ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff38ed9d640 1 -- 172.21.15.64:0/2474723998 reap_dead start 2024-09-17T15:49:59.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff38ed9d640 1 -- 172.21.15.64:0/2474723998 reap_dead start 2024-09-17T15:49:59.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff394905640 1 -- 172.21.15.64:0/2474723998 shutdown_connections 2024-09-17T15:49:59.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff394905640 1 -- 172.21.15.64:0/2474723998 >> 172.21.15.64:0/2474723998 conn(0x7ff39006daa0 msgr2=0x7ff3900a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:59.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff394905640 1 -- 172.21.15.64:0/2474723998 shutdown_connections 2024-09-17T15:49:59.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.680+0000 7ff394905640 1 -- 172.21.15.64:0/2474723998 wait complete. 2024-09-17T15:49:59.721 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:49:59.721 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-17T15:49:59.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-17T15:49:59.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.821+0000 7f22c71f8640 10 monclient: get_monmap_and_config 2024-09-17T15:49:59.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.821+0000 7f22c71f8640 10 monclient: build_initial_monmap 2024-09-17T15:49:59.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.822+0000 7f22c71f8640 10 monclient: monmap: 2024-09-17T15:49:59.824 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:59.824 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.824 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:49:59.822833+0000 2024-09-17T15:49:59.824 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:49:59.822833+0000 2024-09-17T15:49:59.824 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:59.824 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.824 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:59.824 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:59.824 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:59.825 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:59.825 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:59.825 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:59.825 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:59.825 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:59.825 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:59.825 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.822+0000 7f22c71f8640 1 Processor -- start 2024-09-17T15:49:59.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.822+0000 7f22c71f8640 1 -- start start 2024-09-17T15:49:59.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.822+0000 7f22c71f8640 10 monclient: init 2024-09-17T15:49:59.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22c71f8640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:59.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22c71f8640 10 monclient: _add_conns ranks=[5,8,3,0,1,6,7,4,2] 2024-09-17T15:49:59.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22c71f8640 10 monclient(hunting): picked mon.noname-h con 0x7f22c00a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:49:59.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22c71f8640 10 monclient(hunting): picked mon.noname-e con 0x7f22c0135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:49:59.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22c71f8640 10 monclient(hunting): picked mon.noname-g con 0x7f22c0135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:49:59.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22c71f8640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22c009d6f0 con 0x7f22c0135f70 2024-09-17T15:49:59.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22c71f8640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22c009d9d0 con 0x7f22c00a5520 2024-09-17T15:49:59.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22c71f8640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22c009dcb0 con 0x7f22c0135b50 2024-09-17T15:49:59.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22c71f8640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22c71f8640 10 monclient(hunting): authenticate will time out at 997.498358s 2024-09-17T15:49:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bffff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f22c0135b50 0x7f22c013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:58118/0 (socket says 172.21.15.64:58118) 2024-09-17T15:49:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bffff640 1 -- 172.21.15.64:0/4207184566 learned_addr learned my addr 172.21.15.64:0/4207184566 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 1 -- 172.21.15.64:0/4207184566 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1967997674 0 0) 0x7f22c009dcb0 con 0x7f22c0135b50 2024-09-17T15:49:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient(hunting): my global_id is 5732 2024-09-17T15:49:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 1 -- 172.21.15.64:0/4207184566 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22ac003520 con 0x7f22c0135b50 2024-09-17T15:49:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 1 -- 172.21.15.64:0/4207184566 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2933459596 0 0) 0x7f22c009d6f0 con 0x7f22c0135f70 2024-09-17T15:49:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient(hunting): my global_id is 114834 2024-09-17T15:49:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 1 -- 172.21.15.64:0/4207184566 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22c009dcb0 con 0x7f22c0135f70 2024-09-17T15:49:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 1 -- 172.21.15.64:0/4207184566 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1724410006 0 0) 0x7f22c009d9d0 con 0x7f22c00a5520 2024-09-17T15:49:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient(hunting): my global_id is 124223 2024-09-17T15:49:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 1 -- 172.21.15.64:0/4207184566 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22c009d6f0 con 0x7f22c00a5520 2024-09-17T15:49:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 1 -- 172.21.15.64:0/4207184566 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2216538003 0 0) 0x7f22ac003520 con 0x7f22c0135b50 2024-09-17T15:49:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 1 -- 172.21.15.64:0/4207184566 >> v1:172.21.15.100:6791/0 conn(0x7f22c00a5520 legacy=0x7f22c00a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 1 -- 172.21.15.64:0/4207184566 >> v1:172.21.15.100:6790/0 conn(0x7f22c0135f70 legacy=0x7f22c01415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:49:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 1 monclient: found mon.noname-e 2024-09-17T15:49:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:49:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 1 -- 172.21.15.64:0/4207184566 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c009ccd0 con 0x7f22c0135b50 2024-09-17T15:49:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient: _finish_auth 0 2024-09-17T15:49:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:49:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22c71f8640 5 monclient: authenticate success, global_id 5732 2024-09-17T15:49:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 1 -- 172.21.15.64:0/4207184566 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f22a8001830 con 0x7f22c0135b50 2024-09-17T15:49:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22c71f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:49:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.823+0000 7f22bf7fe640 10 monclient: dump: 2024-09-17T15:49:59.838 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.838 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.838 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.838 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.838 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.838 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.838 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.838 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.839 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.839 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.839 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.839 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.839 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.839 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.839 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.839 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22c71f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:59.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22bf7fe640 1 -- 172.21.15.64:0/4207184566 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f22a8001a50 con 0x7f22c0135b50 2024-09-17T15:49:59.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22bf7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:59.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22c71f8640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:59.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22c71f8640 4 set_mon_vals no callback set 2024-09-17T15:49:59.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22bf7fe640 1 -- 172.21.15.64:0/4207184566 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f22a8004be0 con 0x7f22c0135b50 2024-09-17T15:49:59.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22bf7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:49:59.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22bf7fe640 10 monclient: dump: 2024-09-17T15:49:59.840 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.840 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.840 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.840 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.840 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.840 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.841 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.841 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.841 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.841 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.841 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.841 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.841 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.841 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.841 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.841 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22c71f8640 10 monclient: shutdown 2024-09-17T15:49:59.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22c71f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:59.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22c71f8640 1 -- 172.21.15.64:0/4207184566 >> v1:172.21.15.64:6793/0 conn(0x7f22c0135b50 legacy=0x7f22c013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22c71f8640 1 -- 172.21.15.64:0/4207184566 shutdown_connections 2024-09-17T15:49:59.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22c71f8640 1 -- 172.21.15.64:0/4207184566 >> 172.21.15.64:0/4207184566 conn(0x7f22c006db30 msgr2=0x7f22c00ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:59.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22c71f8640 1 -- 172.21.15.64:0/4207184566 shutdown_connections 2024-09-17T15:49:59.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.824+0000 7f22c71f8640 1 -- 172.21.15.64:0/4207184566 wait complete. 2024-09-17T15:49:59.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.825+0000 7f22c71f8640 10 monclient: build_initial_monmap 2024-09-17T15:49:59.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.825+0000 7f22c71f8640 10 monclient: monmap: 2024-09-17T15:49:59.842 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:59.842 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:59.842 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.825+0000 7f22c71f8640 1 Processor -- start 2024-09-17T15:49:59.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.825+0000 7f22c71f8640 1 -- start start 2024-09-17T15:49:59.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.825+0000 7f22c71f8640 10 monclient: init 2024-09-17T15:49:59.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22c71f8640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:59.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22c71f8640 10 monclient: _add_conns ranks=[0,8,1,4,5,7,6,3,2] 2024-09-17T15:49:59.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22c71f8640 10 monclient(hunting): picked mon.noname-a con 0x7f22c00a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:49:59.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22c71f8640 10 monclient(hunting): picked mon.noname-e con 0x7f22c0135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:49:59.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22c71f8640 10 monclient(hunting): picked mon.noname-f con 0x7f22c0135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:49:59.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22c71f8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22c01347d0 con 0x7f22c00a5520 2024-09-17T15:49:59.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22c71f8640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22c009fed0 con 0x7f22c0135f70 2024-09-17T15:49:59.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22c71f8640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22c00a01b0 con 0x7f22c0135b50 2024-09-17T15:49:59.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22c71f8640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:59.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22c71f8640 10 monclient(hunting): authenticate will time out at 997.501329s 2024-09-17T15:49:59.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bffff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f22c0135b50 0x7f22c01320b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:58130/0 (socket says 172.21.15.64:58130) 2024-09-17T15:49:59.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22c4f94640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f22c00a5520 0x7f22c01319a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35178/0 (socket says 172.21.15.64:35178) 2024-09-17T15:49:59.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bffff640 1 -- 172.21.15.64:0/413006651 learned_addr learned my addr 172.21.15.64:0/413006651 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:59.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3681107042 0 0) 0x7f22c009d6f0 con 0x7f22c00a5520 2024-09-17T15:49:59.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): my global_id is 124137 2024-09-17T15:49:59.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22a0003520 con 0x7f22c00a5520 2024-09-17T15:49:59.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 47602348 0 0) 0x7f22c00a01b0 con 0x7f22c0135b50 2024-09-17T15:49:59.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): my global_id is 5741 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22c009d6f0 con 0x7f22c0135b50 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1930321983 0 0) 0x7f22c009fed0 con 0x7f22c0135f70 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): my global_id is 5860 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22c00a01b0 con 0x7f22c0135f70 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 445214015 0 0) 0x7f22a0003520 con 0x7f22c00a5520 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22c009fed0 con 0x7f22c00a5520 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1764279690 0 0) 0x7f22c009d6f0 con 0x7f22c0135b50 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22a0003520 con 0x7f22c0135b50 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f22b0001b10 con 0x7f22c00a5520 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): dump: 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.847 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.848 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.848 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.848 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.826+0000 7f22bd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:49:59.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f22a8004b20 con 0x7f22c0135b50 2024-09-17T15:49:59.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:49:59.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient(hunting): dump: 2024-09-17T15:49:59.848 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.848 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.848 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.848 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.848 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.849 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.849 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.849 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.849 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.849 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.849 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.849 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.849 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.849 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.849 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1002121890 0 0) 0x7f22c00a01b0 con 0x7f22c0135f70 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22c009d6f0 con 0x7f22c0135f70 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f22b8001b50 con 0x7f22c0135f70 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient(hunting): dump: 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.850 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.851 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.851 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.851 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.851 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.851 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.851 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.851 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.851 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.851 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3427710929 0 0) 0x7f22c009fed0 con 0x7f22c00a5520 2024-09-17T15:49:59.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 >> v1:172.21.15.64:6793/0 conn(0x7f22c0135b50 legacy=0x7f22c01320b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 >> v1:172.21.15.100:6789/0 conn(0x7f22c0135f70 legacy=0x7f22c01327c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient: _finish_hunting 0 2024-09-17T15:49:59.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 1 monclient: found mon.a 2024-09-17T15:49:59.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:59.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:49:59.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c00a0770 con 0x7f22c00a5520 2024-09-17T15:49:59.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient: _finish_auth 0 2024-09-17T15:49:59.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:49:59.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:59.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22c71f8640 5 monclient: authenticate success, global_id 124137 2024-09-17T15:49:59.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22c71f8640 10 monclient: _renew_subs 2024-09-17T15:49:59.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22c71f8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:49:59.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22c71f8640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22c0134cb0 con 0x7f22c00a5520 2024-09-17T15:49:59.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22c71f8640 10 monclient: _renew_subs 2024-09-17T15:49:59.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22c71f8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22c71f8640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f22c009eaf0 con 0x7f22c00a5520 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f22b00038c0 con 0x7f22c00a5520 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f22b0003e70 con 0x7f22c00a5520 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22beffd640 4 set_mon_vals no callback set 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.827+0000 7f22bd7fa640 10 monclient: dump: 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.853 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.854 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22c71f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22c71f8640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f22a40065c0 con 0x7f22a4002860 2024-09-17T15:49:59.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f22b0004010 con 0x7f22c00a5520 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(194..194 src has 1..194) ==== 3898+0+0 (unknown 3438653403 0 0) 0x7f22b004ebb0 con 0x7f22c00a5520 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 377159674 0 0) 0x7f22a40065c0 con 0x7f22a4002860 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 10 monclient(hunting): my global_id is 5578 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22c009fed0 con 0x7f22a4002860 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22c5795640 10 monclient: get_auth_request con 0x7f22a0040cc0 auth_method 0 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3011814049 0 0) 0x7f22c009fed0 con 0x7f22a4002860 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22a40065c0 con 0x7f22a4002860 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f22a80051e0 con 0x7f22a4002860 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:49:59.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.828+0000 7f22bd7fa640 10 monclient: dump: 2024-09-17T15:49:59.856 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.856 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.856 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.856 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.856 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.856 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.856 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.856 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.856 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.856 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.856 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.856 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.856 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.857 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.857 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.857 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.829+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3602639013 0 0) 0x7f22a40065c0 con 0x7f22a4002860 2024-09-17T15:49:59.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.829+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.100:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22a400a160 con 0x7f22a4002860 2024-09-17T15:49:59.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.829+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f22a8005310 con 0x7f22a4002860 2024-09-17T15:49:59.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.829+0000 7f22bd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:59.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.829+0000 7f22bd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:59.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.829+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 >> v1:172.21.15.100:6792/0 conn(0x7f22a4002860 legacy=0x7f22a4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.829+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 >> v1:172.21.15.100:6792/0 conn(0x7f22a4002860 legacy=0x7f22a4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:59.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.835+0000 7f22c71f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:59.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.835+0000 7f22c71f8640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f22a4009e50 con 0x7f22a4009110 2024-09-17T15:49:59.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1368756255 0 0) 0x7f22a4009e50 con 0x7f22a4009110 2024-09-17T15:49:59.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 10 monclient(hunting): my global_id is 5587 2024-09-17T15:49:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22a40065c0 con 0x7f22a4009110 2024-09-17T15:49:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2275516466 0 0) 0x7f22a40065c0 con 0x7f22a4009110 2024-09-17T15:49:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22a4009e50 con 0x7f22a4009110 2024-09-17T15:49:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f22a80057f0 con 0x7f22a4009110 2024-09-17T15:49:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:49:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 10 monclient: dump: 2024-09-17T15:49:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.860 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.861 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.861 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.861 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.861 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.861 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.861 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.861 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.861 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.861 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.861 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.861 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.862 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3280245743 0 0) 0x7f22a4009e50 con 0x7f22a4009110 2024-09-17T15:49:59.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.836+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 --> v1:172.21.15.100:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f22a400c3f0 con 0x7f22a4009110 2024-09-17T15:49:59.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 4134292899) 0x7f22a8007cd0 con 0x7f22a4009110 2024-09-17T15:49:59.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22bd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:59.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22bd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:59.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 >> v1:172.21.15.100:6792/0 conn(0x7f22a4009110 legacy=0x7f22a40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22bd7fa640 1 -- 172.21.15.64:0/413006651 >> v1:172.21.15.100:6792/0 conn(0x7f22a4009110 legacy=0x7f22a40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:59.863 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:49:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:49:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:49:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:49:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:49:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 256471, 2024-09-17T15:49:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:49:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:49:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:49:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:49:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:49:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:49:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:49:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:49:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:49:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:49:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:49:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:49:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:49:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:49:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:49:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:49:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:49:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:49:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:49:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:49:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:49:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:49:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:49:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:49:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:49:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:49:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:49:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:49:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:49:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:49:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:49:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:49:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:49:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:49:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:49:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:49:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:49:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:49:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:49:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:49:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:49:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:49:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:49:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:49:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:49:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:49:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:49:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:49:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:49:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:49:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:49:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:49:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 4 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": [ 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:49:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:49:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:49:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:49:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:49:59.883 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:49:59.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22c71f8640 1 -- 172.21.15.64:0/413006651 >> v1:172.21.15.100:6812/4210963484 conn(0x7f22a0040cc0 legacy=0x7f22a0043180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22c71f8640 10 monclient: shutdown 2024-09-17T15:49:59.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22c71f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:59.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22c71f8640 1 -- 172.21.15.64:0/413006651 >> v1:172.21.15.64:6789/0 conn(0x7f22c00a5520 legacy=0x7f22c01319a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22c5795640 1 -- 172.21.15.64:0/413006651 reap_dead start 2024-09-17T15:49:59.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22c5795640 1 -- 172.21.15.64:0/413006651 reap_dead start 2024-09-17T15:49:59.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22c71f8640 1 -- 172.21.15.64:0/413006651 shutdown_connections 2024-09-17T15:49:59.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22c71f8640 1 -- 172.21.15.64:0/413006651 >> 172.21.15.64:0/413006651 conn(0x7f22c006db30 msgr2=0x7f22c012c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:59.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22c71f8640 1 -- 172.21.15.64:0/413006651 shutdown_connections 2024-09-17T15:49:59.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.837+0000 7f22c71f8640 1 -- 172.21.15.64:0/413006651 wait complete. 2024-09-17T15:49:59.885 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:49:59.885 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:49:59.885 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:49:59.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.985+0000 7fcc03789640 10 monclient: get_monmap_and_config 2024-09-17T15:49:59.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.985+0000 7fcc03789640 10 monclient: build_initial_monmap 2024-09-17T15:49:59.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.985+0000 7fcc03789640 10 monclient: monmap: 2024-09-17T15:49:59.987 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:49:59.987 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.987 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:49:59.986044+0000 2024-09-17T15:49:59.987 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:49:59.986044+0000 2024-09-17T15:49:59.987 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:59.987 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.987 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:49:59.987 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:49:59.987 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:49:59.987 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:49:59.988 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:49:59.988 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:49:59.988 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:49:59.988 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:49:59.988 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:49:59.988 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.985+0000 7fcc03789640 1 Processor -- start 2024-09-17T15:49:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc03789640 1 -- start start 2024-09-17T15:49:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc03789640 10 monclient: init 2024-09-17T15:49:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc03789640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc03789640 10 monclient: _add_conns ranks=[6,5,1,0,7,8,2,3,4] 2024-09-17T15:49:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc03789640 10 monclient(hunting): picked mon.noname-d con 0x7fcbfc0a54b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:49:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc03789640 10 monclient(hunting): picked mon.noname-h con 0x7fcbfc135c80 addr v1:172.21.15.100:6791/0 2024-09-17T15:49:59.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc03789640 10 monclient(hunting): picked mon.noname-f con 0x7fcbfc1360a0 addr v1:172.21.15.100:6789/0 2024-09-17T15:49:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc03789640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbfc0aacb0 con 0x7fcbfc1360a0 2024-09-17T15:49:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc03789640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbfc09d780 con 0x7fcbfc135c80 2024-09-17T15:49:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc03789640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbfc09da60 con 0x7fcbfc0a54b0 2024-09-17T15:49:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc03789640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc03789640 10 monclient(hunting): authenticate will time out at 997.661546s 2024-09-17T15:49:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc01525640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fcbfc0a54b0 0x7fcbfc0a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:44036/0 (socket says 172.21.15.64:44036) 2024-09-17T15:49:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcc01525640 1 -- 172.21.15.64:0/2563088888 learned_addr learned my addr 172.21.15.64:0/2563088888 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:49:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcbfbfff640 1 -- 172.21.15.64:0/2563088888 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 399189513 0 0) 0x7fcbfc09da60 con 0x7fcbfc0a54b0 2024-09-17T15:49:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcbfbfff640 10 monclient(hunting): _negotiate 2024-09-17T15:49:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcbfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcbfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:59.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcbfbfff640 10 monclient(hunting): my global_id is 124170 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.986+0000 7fcbfbfff640 1 -- 172.21.15.64:0/2563088888 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbe4003520 con 0x7fcbfc0a54b0 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 1 -- 172.21.15.64:0/2563088888 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 625800825 0 0) 0x7fcbe4003520 con 0x7fcbfc0a54b0 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 1 -- 172.21.15.64:0/2563088888 >> v1:172.21.15.100:6791/0 conn(0x7fcbfc135c80 legacy=0x7fcbfc13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 1 -- 172.21.15.64:0/2563088888 >> v1:172.21.15.100:6789/0 conn(0x7fcbfc1360a0 legacy=0x7fcbfc141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 10 monclient: _finish_hunting 0 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 1 monclient: found mon.noname-d 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 1 -- 172.21.15.64:0/2563088888 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc09dd40 con 0x7fcbfc0a54b0 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 10 monclient: _finish_auth 0 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 1 -- 172.21.15.64:0/2563088888 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcbf4001830 con 0x7fcbfc0a54b0 2024-09-17T15:49:59.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcc03789640 5 monclient: authenticate success, global_id 124170 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcc03789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 10 monclient: dump: 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:49:59.998 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcc03789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 1 -- 172.21.15.64:0/2563088888 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcbf40019d0 con 0x7fcbfc0a54b0 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcc03789640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 1 -- 172.21.15.64:0/2563088888 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcbf4004be0 con 0x7fcbfc0a54b0 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcc03789640 4 set_mon_vals no callback set 2024-09-17T15:49:59.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:50:00.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcbfbfff640 10 monclient: dump: 2024-09-17T15:50:00.000 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:00.000 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:00.000 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:00.000 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:00.000 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:00.000 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:00.000 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:00.000 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:00.000 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:00.000 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:00.000 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:00.000 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcc03789640 10 monclient: shutdown 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcc03789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcc03789640 1 -- 172.21.15.64:0/2563088888 >> v1:172.21.15.64:6792/0 conn(0x7fcbfc0a54b0 legacy=0x7fcbfc0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcc03789640 1 -- 172.21.15.64:0/2563088888 shutdown_connections 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcc03789640 1 -- 172.21.15.64:0/2563088888 >> 172.21.15.64:0/2563088888 conn(0x7fcbfc06daa0 msgr2=0x7fcbfc0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcc03789640 1 -- 172.21.15.64:0/2563088888 shutdown_connections 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.987+0000 7fcc03789640 1 -- 172.21.15.64:0/2563088888 wait complete. 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.988+0000 7fcc03789640 10 monclient: build_initial_monmap 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.988+0000 7fcc03789640 10 monclient: monmap: 2024-09-17T15:50:00.001 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:00.002 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.988+0000 7fcc03789640 1 Processor -- start 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.988+0000 7fcc03789640 1 -- start start 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.988+0000 7fcc03789640 10 monclient: init 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcc03789640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcc03789640 10 monclient: _add_conns ranks=[1,6,3,8,7,0,2,5,4] 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcc03789640 10 monclient(hunting): picked mon.noname-f con 0x7fcbfc0a54b0 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcc03789640 10 monclient(hunting): picked mon.noname-d con 0x7fcbfc135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcc03789640 10 monclient(hunting): picked mon.noname-g con 0x7fcbfc1360a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcc03789640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbfc0a0480 con 0x7fcbfc0a54b0 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcc03789640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbfc1cebf0 con 0x7fcbfc1360a0 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcc03789640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbfc0a0760 con 0x7fcbfc135c80 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcc03789640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:00.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcc03789640 10 monclient(hunting): authenticate will time out at 997.664437s 2024-09-17T15:50:00.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcc00d24640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fcbfc135c80 0x7fcbfc1ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:44046/0 (socket says 172.21.15.64:44046) 2024-09-17T15:50:00.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcc00d24640 1 -- 172.21.15.64:0/1239770248 learned_addr learned my addr 172.21.15.64:0/1239770248 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:00.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2039380425 0 0) 0x7fcbfc0a0760 con 0x7fcbfc135c80 2024-09-17T15:50:00.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:50:00.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:00.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:00.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): my global_id is 124179 2024-09-17T15:50:00.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbe8003520 con 0x7fcbfc135c80 2024-09-17T15:50:00.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1953494707 0 0) 0x7fcbfc0a0480 con 0x7fcbfc0a54b0 2024-09-17T15:50:00.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:50:00.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:00.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:00.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): my global_id is 5878 2024-09-17T15:50:00.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbfc0a0760 con 0x7fcbfc0a54b0 2024-09-17T15:50:00.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 680494763 0 0) 0x7fcbfc1cebf0 con 0x7fcbfc1360a0 2024-09-17T15:50:00.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:50:00.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:00.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:00.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): my global_id is 114843 2024-09-17T15:50:00.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbfc0a0480 con 0x7fcbfc1360a0 2024-09-17T15:50:00.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 76213682 0 0) 0x7fcbe8003520 con 0x7fcbfc135c80 2024-09-17T15:50:00.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbfc1cebf0 con 0x7fcbfc135c80 2024-09-17T15:50:00.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcbf0001f50 con 0x7fcbfc135c80 2024-09-17T15:50:00.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:00.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:50:00.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): dump: 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:00.006 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:00.007 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:00.007 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:00.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.989+0000 7fcbf9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:00.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3658559083 0 0) 0x7fcbfc0a0760 con 0x7fcbfc0a54b0 2024-09-17T15:50:00.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbe8003520 con 0x7fcbfc0a54b0 2024-09-17T15:50:00.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcbf40016f0 con 0x7fcbfc0a54b0 2024-09-17T15:50:00.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:00.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:00.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient(hunting): dump: 2024-09-17T15:50:00.007 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:00.007 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:00.007 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:00.007 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3286180682 0 0) 0x7fcbfc0a0480 con 0x7fcbfc1360a0 2024-09-17T15:50:00.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbfc0a0760 con 0x7fcbfc1360a0 2024-09-17T15:50:00.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcbec001830 con 0x7fcbfc1360a0 2024-09-17T15:50:00.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:00.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:50:00.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient(hunting): dump: 2024-09-17T15:50:00.009 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:00.009 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:00.009 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:00.009 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:00.009 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:00.009 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:00.009 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:00.009 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:00.009 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:00.010 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:00.010 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:00.010 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:00.010 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:00.010 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:00.010 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:00.010 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:00.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3090775036 0 0) 0x7fcbfc1cebf0 con 0x7fcbfc135c80 2024-09-17T15:50:00.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 >> v1:172.21.15.100:6790/0 conn(0x7fcbfc1360a0 legacy=0x7fcbfc1d6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:00.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 >> v1:172.21.15.100:6789/0 conn(0x7fcbfc0a54b0 legacy=0x7fcbfc1cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:00.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:50:00.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 1 monclient: found mon.d 2024-09-17T15:50:00.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc132c30 con 0x7fcbfc135c80 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcc03789640 5 monclient: authenticate success, global_id 124179 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcc03789640 10 monclient: _renew_subs 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcc03789640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcc03789640 1 -- 172.21.15.64:0/1239770248 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcbfc0a01a0 con 0x7fcbfc135c80 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcbf0003c40 con 0x7fcbfc135c80 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcbf0004170 con 0x7fcbfc135c80 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:00.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbf9ffb640 10 monclient: dump: 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:00.012 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:00.013 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:00.013 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:00.013 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:00.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcc03789640 10 monclient: _renew_subs 2024-09-17T15:50:00.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcbfb7fe640 4 set_mon_vals no callback set 2024-09-17T15:50:00.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcc03789640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:50:00.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.990+0000 7fcc03789640 1 -- 172.21.15.64:0/1239770248 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fcbfc1d7750 con 0x7fcbfc135c80 2024-09-17T15:50:00.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.991+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fcbf0004310 con 0x7fcbfc135c80 2024-09-17T15:50:00.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.991+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(194..194 src has 1..194) ==== 3898+0+0 (unknown 3438653403 0 0) 0x7fcbf004f090 con 0x7fcbfc135c80 2024-09-17T15:50:00.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.991+0000 7fcc03789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:00.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.991+0000 7fcc03789640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:00.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.991+0000 7fcc03789640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:50:00.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.991+0000 7fcc03789640 1 -- 172.21.15.64:0/1239770248 --> v1:172.21.15.64:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbfc0ac2a0 con 0x7fcbfc135c80 2024-09-17T15:50:00.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.991+0000 7fcc01525640 10 monclient: get_auth_request con 0x7fcbe8040d70 auth_method 0 2024-09-17T15:50:00.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.996+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.6 v1:172.21.15.64:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fcbfc0ac2a0 con 0x7fcbfc135c80 2024-09-17T15:50:00.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.996+0000 7fcbf9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:00.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:49:59.996+0000 7fcbf9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:00.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.255+0000 7fcc03789640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:50:00.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.255+0000 7fcc03789640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:50:00.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.255+0000 7fcc03789640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:50:00.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.255+0000 7fcc03789640 1 -- 172.21.15.64:0/1239770248 --> v1:172.21.15.64:6792/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fcbfc0694b0 con 0x7fcbfc135c80 2024-09-17T15:50:00.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.257+0000 7fcbf9ffb640 1 -- 172.21.15.64:0/1239770248 <== mon.6 v1:172.21.15.64:6792/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fcbfc0694b0 con 0x7fcbfc135c80 2024-09-17T15:50:00.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.257+0000 7fcbf9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:50:00.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.257+0000 7fcbf9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:00.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=1,health=10,kv=12,logm=57} crc {auth=769103218,config=1747058506,health=3235492580,kv=1246312931,logm=1599915127}) 2024-09-17T15:50:00.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=11,kv=11,logm=55} crc {auth=1299389711,config=2586702516,health=785757143,kv=1436989955,logm=4142291693}) 2024-09-17T15:50:00.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=1,health=10,kv=12,logm=57} crc {auth=769103218,config=1747058506,health=3235492580,kv=1246312931,logm=1599915127}) 2024-09-17T15:50:00.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=12,kv=10,logm=55} crc {auth=867121501,config=2586702516,health=2868366208,kv=3466664689,logm=1641920074}) 2024-09-17T15:50:00.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=1,health=10,kv=12,logm=57} crc {auth=769103218,config=1747058506,health=3235492580,kv=1246312931,logm=1599915127}) 2024-09-17T15:50:00.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=19,config=2,health=12,kv=11,logm=56} crc {auth=3576691530,config=2586702516,health=73906507,kv=4080546736,logm=3803056504}) 2024-09-17T15:50:00.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=1,health=10,kv=12,logm=57} crc {auth=769103218,config=1747058506,health=3235492580,kv=1246312931,logm=1599915127}) 2024-09-17T15:50:00.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=3986871827,config=2586702516,health=3507680350,kv=1051266338,logm=3124342092}) 2024-09-17T15:50:00.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=1,health=10,kv=12,logm=57} crc {auth=769103218,config=1747058506,health=3235492580,kv=1246312931,logm=1599915127}) 2024-09-17T15:50:00.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=21,config=2,health=11,kv=12,logm=54} crc {auth=1826580072,config=2586702516,health=214281541,kv=1246312931,logm=264879255}) 2024-09-17T15:50:00.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=1,health=10,kv=12,logm=57} crc {auth=769103218,config=1747058506,health=3235492580,kv=1246312931,logm=1599915127}) 2024-09-17T15:50:00.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=19,config=2,health=10,kv=12,logm=57} crc {auth=573266010,config=2586702516,health=120715245,kv=1246312931,logm=2602531130}) 2024-09-17T15:50:00.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=1,health=10,kv=12,logm=57} crc {auth=769103218,config=1747058506,health=3235492580,kv=1246312931,logm=1599915127}) 2024-09-17T15:50:00.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=18,config=2,health=11,kv=12,logm=57} crc {auth=514645802,config=2586702516,health=3649455617,kv=1246312931,logm=1487515679}) 2024-09-17T15:50:00.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=1,health=10,kv=12,logm=57} crc {auth=769103218,config=1747058506,health=3235492580,kv=1246312931,logm=1599915127}) 2024-09-17T15:50:00.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.256+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=20,config=2,health=10,kv=10,logm=58} crc {auth=569580253,config=2586702516,health=2380522673,kv=1970731663,logm=1951837774}) 2024-09-17T15:50:00.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.257+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939942332}) 2024-09-17T15:50:00.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365251896}) 2024-09-17T15:50:00.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939942332}) 2024-09-17T15:50:00.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2367685810}) 2024-09-17T15:50:00.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939942332}) 2024-09-17T15:50:00.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2272973244}) 2024-09-17T15:50:00.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939942332}) 2024-09-17T15:50:00.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1328114552}) 2024-09-17T15:50:00.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939942332}) 2024-09-17T15:50:00.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=683460426}) 2024-09-17T15:50:00.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939942332}) 2024-09-17T15:50:00.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1322947421}) 2024-09-17T15:50:00.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939942332}) 2024-09-17T15:50:00.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2255640605}) 2024-09-17T15:50:00.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939942332}) 2024-09-17T15:50:00.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.258+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3759084813}) 2024-09-17T15:50:00.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618346023}) 2024-09-17T15:50:00.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1567100494}) 2024-09-17T15:50:00.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618346023}) 2024-09-17T15:50:00.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4086250400}) 2024-09-17T15:50:00.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618346023}) 2024-09-17T15:50:00.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1513961307}) 2024-09-17T15:50:00.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618346023}) 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1427112078}) 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618346023}) 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1756182867}) 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618346023}) 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3228593571}) 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618346023}) 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2877707887}) 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618346023}) 2024-09-17T15:50:00.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.259+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3223738109}) 2024-09-17T15:50:00.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.260+0000 7fcc03789640 1 -- 172.21.15.64:0/1239770248 >> v1:172.21.15.100:6812/4210963484 conn(0x7fcbe8040d70 legacy=0x7fcbe8043230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:00.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.260+0000 7fcc03789640 10 monclient: shutdown 2024-09-17T15:50:00.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.260+0000 7fcc03789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:00.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.260+0000 7fcc03789640 1 -- 172.21.15.64:0/1239770248 >> v1:172.21.15.64:6792/0 conn(0x7fcbfc135c80 legacy=0x7fcbfc1ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:00.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.260+0000 7fcc03789640 1 -- 172.21.15.64:0/1239770248 shutdown_connections 2024-09-17T15:50:00.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.260+0000 7fcc03789640 1 -- 172.21.15.64:0/1239770248 >> 172.21.15.64:0/1239770248 conn(0x7fcbfc06daa0 msgr2=0x7fcbfc133930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:00.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.260+0000 7fcc03789640 1 -- 172.21.15.64:0/1239770248 shutdown_connections 2024-09-17T15:50:00.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:00.260+0000 7fcc03789640 1 -- 172.21.15.64:0/1239770248 wait complete. 2024-09-17T15:50:00.281 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:50:00.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1395712705}) 2024-09-17T15:50:00.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2929630219}) 2024-09-17T15:50:00.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1395712705}) 2024-09-17T15:50:00.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=688320865}) 2024-09-17T15:50:00.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1395712705}) 2024-09-17T15:50:00.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2283354677}) 2024-09-17T15:50:00.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1395712705}) 2024-09-17T15:50:00.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1865892713}) 2024-09-17T15:50:00.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1395712705}) 2024-09-17T15:50:00.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1191661716}) 2024-09-17T15:50:00.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1395712705}) 2024-09-17T15:50:00.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2298189685}) 2024-09-17T15:50:00.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1395712705}) 2024-09-17T15:50:00.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1861309287}) 2024-09-17T15:50:00.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1395712705}) 2024-09-17T15:50:00.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.260+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3040354870}) 2024-09-17T15:50:00.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669570794}) 2024-09-17T15:50:00.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377246165}) 2024-09-17T15:50:00.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669570794}) 2024-09-17T15:50:00.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673267311}) 2024-09-17T15:50:00.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669570794}) 2024-09-17T15:50:00.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1856846265}) 2024-09-17T15:50:00.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669570794}) 2024-09-17T15:50:00.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2837174933}) 2024-09-17T15:50:00.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669570794}) 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1952355843}) 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669570794}) 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2903955781}) 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669570794}) 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=403896695}) 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669570794}) 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.262+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=303129409}) 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2445202251}) 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2535280815}) 2024-09-17T15:50:00.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2445202251}) 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2479808321}) 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2445202251}) 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3586139328}) 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2445202251}) 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=586209543}) 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2445202251}) 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=414392088}) 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2445202251}) 2024-09-17T15:50:00.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=575529282}) 2024-09-17T15:50:00.297 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.297 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2445202251}) 2024-09-17T15:50:00.297 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2834271706}) 2024-09-17T15:50:00.297 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.297 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2445202251}) 2024-09-17T15:50:00.297 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.263+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=224074786}) 2024-09-17T15:50:00.297 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.297 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149818007}) 2024-09-17T15:50:00.297 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373106041}) 2024-09-17T15:50:00.297 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.297 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149818007}) 2024-09-17T15:50:00.297 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1507865614}) 2024-09-17T15:50:00.297 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.298 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149818007}) 2024-09-17T15:50:00.298 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2135159972}) 2024-09-17T15:50:00.298 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.298 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149818007}) 2024-09-17T15:50:00.298 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2797777502}) 2024-09-17T15:50:00.298 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.298 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149818007}) 2024-09-17T15:50:00.298 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=448013757}) 2024-09-17T15:50:00.298 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.298 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149818007}) 2024-09-17T15:50:00.298 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3574553554}) 2024-09-17T15:50:00.298 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.298 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149818007}) 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=193743886}) 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149818007}) 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.264+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3175713160}) 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1382275165}) 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068087005}) 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1382275165}) 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=622892435}) 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1382275165}) 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4126577363}) 2024-09-17T15:50:00.299 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1382275165}) 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1123677347}) 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1382275165}) 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1546248009}) 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1382275165}) 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2061164309}) 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1382275165}) 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1917546667}) 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1382275165}) 2024-09-17T15:50:00.300 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.265+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3900257897}) 2024-09-17T15:50:00.301 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.266+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.301 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.266+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705359684}) 2024-09-17T15:50:00.301 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205882301}) 2024-09-17T15:50:00.301 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.301 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705359684}) 2024-09-17T15:50:00.301 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3460514670}) 2024-09-17T15:50:00.301 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.301 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705359684}) 2024-09-17T15:50:00.301 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1290803521}) 2024-09-17T15:50:00.301 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.301 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705359684}) 2024-09-17T15:50:00.301 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1239015923}) 2024-09-17T15:50:00.301 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.302 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705359684}) 2024-09-17T15:50:00.302 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3028822166}) 2024-09-17T15:50:00.302 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.302 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705359684}) 2024-09-17T15:50:00.302 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2767687264}) 2024-09-17T15:50:00.302 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.302 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705359684}) 2024-09-17T15:50:00.302 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=669099881}) 2024-09-17T15:50:00.302 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.302 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705359684}) 2024-09-17T15:50:00.302 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.267+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=16587935}) 2024-09-17T15:50:00.302 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.302 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634369432}) 2024-09-17T15:50:00.303 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=194684347}) 2024-09-17T15:50:00.303 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.303 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634369432}) 2024-09-17T15:50:00.303 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3787674041}) 2024-09-17T15:50:00.303 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.303 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634369432}) 2024-09-17T15:50:00.303 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3076827365}) 2024-09-17T15:50:00.303 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.303 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634369432}) 2024-09-17T15:50:00.303 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=692612549}) 2024-09-17T15:50:00.303 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.303 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634369432}) 2024-09-17T15:50:00.303 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1041941846}) 2024-09-17T15:50:00.304 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.304 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634369432}) 2024-09-17T15:50:00.304 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2842435270}) 2024-09-17T15:50:00.304 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.304 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634369432}) 2024-09-17T15:50:00.304 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2285657897}) 2024-09-17T15:50:00.304 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.304 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634369432}) 2024-09-17T15:50:00.304 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.268+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2308632915}) 2024-09-17T15:50:00.304 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.304 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2506966143}) 2024-09-17T15:50:00.304 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597985707}) 2024-09-17T15:50:00.304 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2506966143}) 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=196802092}) 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2506966143}) 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2460264522}) 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2506966143}) 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=787725742}) 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2506966143}) 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=556928925}) 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2506966143}) 2024-09-17T15:50:00.305 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.269+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3795534579}) 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.270+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.270+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2506966143}) 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.270+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2278928680}) 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.270+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.270+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2506966143}) 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.270+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3842162728}) 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3586127957}) 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2561341017}) 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3586127957}) 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3479449441}) 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.306 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3586127957}) 2024-09-17T15:50:00.307 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2808190259}) 2024-09-17T15:50:00.307 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.307 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3586127957}) 2024-09-17T15:50:00.307 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1090190368}) 2024-09-17T15:50:00.307 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.307 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3586127957}) 2024-09-17T15:50:00.307 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2157555531}) 2024-09-17T15:50:00.307 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.307 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3586127957}) 2024-09-17T15:50:00.307 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3564272259}) 2024-09-17T15:50:00.307 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.307 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3586127957}) 2024-09-17T15:50:00.307 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=892507187}) 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3586127957}) 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.271+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2034060855}) 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062299040}) 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4066769947}) 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062299040}) 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3747692346}) 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062299040}) 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1555974340}) 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.308 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062299040}) 2024-09-17T15:50:00.309 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1971145181}) 2024-09-17T15:50:00.309 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.309 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062299040}) 2024-09-17T15:50:00.309 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3162477039}) 2024-09-17T15:50:00.309 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.309 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062299040}) 2024-09-17T15:50:00.309 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2743385016}) 2024-09-17T15:50:00.309 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.309 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062299040}) 2024-09-17T15:50:00.309 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1090825816}) 2024-09-17T15:50:00.309 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.309 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062299040}) 2024-09-17T15:50:00.309 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.272+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1636699515}) 2024-09-17T15:50:00.310 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.310 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=224558299,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=2847491211,mon_config_key=137185324,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=1630794215}) 2024-09-17T15:50:00.310 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=21,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=28} crc {logm=3704391057,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1748216916,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=28369507}) 2024-09-17T15:50:00.310 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.310 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=224558299,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=2847491211,mon_config_key=137185324,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=1630794215}) 2024-09-17T15:50:00.310 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=21,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=15,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=31} crc {logm=784369775,mdsmap=1462844619,mgr=2843912955,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=2115134727,mon_config_key=4260132802,monmap=1573330833,nvmeofgw=3140514009,osd_metadata=2749459017,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=1393699051}) 2024-09-17T15:50:00.310 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.310 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=224558299,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=2847491211,mon_config_key=137185324,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=1630794215}) 2024-09-17T15:50:00.310 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=20,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=15,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=31} crc {logm=3291205398,mdsmap=1462844619,mgr=4116046384,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=2115134727,mon_config_key=3198036839,monmap=1573330833,nvmeofgw=3140514009,osd_metadata=4100752439,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=4029128070}) 2024-09-17T15:50:00.310 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.310 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=224558299,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=2847491211,mon_config_key=137185324,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=1630794215}) 2024-09-17T15:50:00.310 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=16,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=6,osd_snap=1,osdmap=35} crc {logm=3690192421,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073260,mgr_metadata=4041113850,mgrstat=3107566979,mon_config_key=1201604950,monmap=3140514009,nvmeofgw=598337686,osd_metadata=4100752439,osd_snap=3823785925,osdmap=1819129303}) 2024-09-17T15:50:00.310 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.311 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=224558299,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=2847491211,mon_config_key=137185324,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=1630794215}) 2024-09-17T15:50:00.311 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=20,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=29} crc {logm=866121638,mdsmap=1462844619,mgr=4278719971,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=2115134727,mon_config_key=2795318339,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=3893215221}) 2024-09-17T15:50:00.311 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.311 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=224558299,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=2847491211,mon_config_key=137185324,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=1630794215}) 2024-09-17T15:50:00.311 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=21,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=28} crc {logm=3704391057,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=2115134727,mon_config_key=2187931326,monmap=1573330833,nvmeofgw=598337686,osd_metadata=195458673,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=1841403980}) 2024-09-17T15:50:00.311 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.311 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=224558299,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=2847491211,mon_config_key=137185324,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=1630794215}) 2024-09-17T15:50:00.311 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=21,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=31} crc {logm=3704391057,mdsmap=2194534611,mgr=232808329,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=3855956556,mon_config_key=3137332142,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=642955184}) 2024-09-17T15:50:00.311 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.311 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=224558299,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=2847491211,mon_config_key=137185324,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=1630794215}) 2024-09-17T15:50:00.311 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.274+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=30} crc {logm=3206750829,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=2254470936,mon_config_key=2807123856,monmap=1573330833,nvmeofgw=1596038039,osd_metadata=3331695128,osd_pg_creating=3006195080,osd_snap=723727393,osdmap=3702672747}) 2024-09-17T15:50:00.311 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.311 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3427204753}) 2024-09-17T15:50:00.312 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2547305892}) 2024-09-17T15:50:00.312 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.312 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3427204753}) 2024-09-17T15:50:00.312 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4107952373}) 2024-09-17T15:50:00.312 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.312 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3427204753}) 2024-09-17T15:50:00.312 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=40225851}) 2024-09-17T15:50:00.312 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.312 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3427204753}) 2024-09-17T15:50:00.312 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3211307330}) 2024-09-17T15:50:00.312 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.312 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3427204753}) 2024-09-17T15:50:00.312 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3606441018}) 2024-09-17T15:50:00.313 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.313 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3427204753}) 2024-09-17T15:50:00.313 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3934208899}) 2024-09-17T15:50:00.313 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.313 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3427204753}) 2024-09-17T15:50:00.313 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=562143526}) 2024-09-17T15:50:00.313 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.313 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3427204753}) 2024-09-17T15:50:00.313 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.275+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1916922636}) 2024-09-17T15:50:00.313 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.313 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2401935020}) 2024-09-17T15:50:00.313 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1567794285}) 2024-09-17T15:50:00.313 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2401935020}) 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1518622748}) 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2401935020}) 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=4231323444}) 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2401935020}) 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1151217903}) 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2401935020}) 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1333646752}) 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2401935020}) 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1105024592}) 2024-09-17T15:50:00.314 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.315 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2401935020}) 2024-09-17T15:50:00.315 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=489270222}) 2024-09-17T15:50:00.315 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.315 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2401935020}) 2024-09-17T15:50:00.315 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.277+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3761089330}) 2024-09-17T15:50:00.315 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.315 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=61} crc {osdmap=436666378}) 2024-09-17T15:50:00.315 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=60} crc {osdmap=3762145437}) 2024-09-17T15:50:00.315 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.315 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=61} crc {osdmap=436666378}) 2024-09-17T15:50:00.315 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=63} crc {osdmap=1736804588}) 2024-09-17T15:50:00.315 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.315 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=61} crc {osdmap=436666378}) 2024-09-17T15:50:00.316 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=63} crc {osdmap=1270245139}) 2024-09-17T15:50:00.316 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.316 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=61} crc {osdmap=436666378}) 2024-09-17T15:50:00.316 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=60} crc {osdmap=1488962954}) 2024-09-17T15:50:00.316 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.316 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=61} crc {osdmap=436666378}) 2024-09-17T15:50:00.316 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=62} crc {osdmap=279591642}) 2024-09-17T15:50:00.316 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.316 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=61} crc {osdmap=436666378}) 2024-09-17T15:50:00.316 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=64} crc {osdmap=1994041844}) 2024-09-17T15:50:00.316 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.316 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=61} crc {osdmap=436666378}) 2024-09-17T15:50:00.316 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=62} crc {osdmap=2562260944}) 2024-09-17T15:50:00.317 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:00.317 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=61} crc {osdmap=436666378}) 2024-09-17T15:50:00.317 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:50:00.279+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=62} crc {osdmap=82991373}) 2024-09-17T15:50:00.531 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:00.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38013]: starting. 2024-09-17T15:50:00.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38013]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-37917 2024-09-17T15:50:00.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38013]: created object 0... 2024-09-17T15:50:00.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38013]: created object 25... 2024-09-17T15:50:00.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38013]: created object 49... 2024-09-17T15:50:00.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38013]: finishing. 2024-09-17T15:50:00.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38013]: shutting down. 2024-09-17T15:50:00.542 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:00.542 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38015]: starting. 2024-09-17T15:50:00.542 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38015]: listing objects. 2024-09-17T15:50:00.542 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38015]: listed object 0... 2024-09-17T15:50:00.542 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38015]: listed object 25... 2024-09-17T15:50:00.542 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38015]: saw 50 objects 2024-09-17T15:50:00.542 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38015]: shutting down. 2024-09-17T15:50:01.282 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-17T15:50:01.282 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:50:01.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.395+0000 7f52d5472640 10 monclient: get_monmap_and_config 2024-09-17T15:50:01.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.395+0000 7f52d5472640 10 monclient: build_initial_monmap 2024-09-17T15:50:01.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.395+0000 7f52d5472640 10 monclient: monmap: 2024-09-17T15:50:01.397 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:01.397 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.397 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:01.396219+0000 2024-09-17T15:50:01.398 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:01.396219+0000 2024-09-17T15:50:01.398 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:01.398 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.398 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:01.398 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:01.398 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:01.398 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:01.398 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:01.398 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:01.398 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:01.398 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:01.398 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:01.399 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.396+0000 7f52d5472640 1 Processor -- start 2024-09-17T15:50:01.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.396+0000 7f52d5472640 1 -- start start 2024-09-17T15:50:01.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.396+0000 7f52d5472640 10 monclient: init 2024-09-17T15:50:01.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.396+0000 7f52d5472640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:01.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.396+0000 7f52d5472640 10 monclient: _add_conns ranks=[7,3,5,0,8,6,2,1,4] 2024-09-17T15:50:01.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.396+0000 7f52d5472640 10 monclient(hunting): picked mon.noname-i con 0x7f52d009ca00 addr v1:172.21.15.100:6792/0 2024-09-17T15:50:01.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.396+0000 7f52d5472640 10 monclient(hunting): picked mon.noname-g con 0x7f52d00a5d10 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:01.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.396+0000 7f52d5472640 10 monclient(hunting): picked mon.noname-h con 0x7f52d0135ba0 addr v1:172.21.15.100:6791/0 2024-09-17T15:50:01.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.396+0000 7f52d5472640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52d0135fe0 con 0x7f52d00a5d10 2024-09-17T15:50:01.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.396+0000 7f52d5472640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52d0136260 con 0x7f52d0135ba0 2024-09-17T15:50:01.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.396+0000 7f52d5472640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52d00ab640 con 0x7f52d009ca00 2024-09-17T15:50:01.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.396+0000 7f52d5472640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:01.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.396+0000 7f52d5472640 10 monclient(hunting): authenticate will time out at 999.071803s 2024-09-17T15:50:01.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52c7fff640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f52d00a5d10 0x7f52d013dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:53320/0 (socket says 172.21.15.64:53320) 2024-09-17T15:50:01.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52c7fff640 1 -- 172.21.15.64:0/1580161515 learned_addr learned my addr 172.21.15.64:0/1580161515 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:01.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ceffd640 1 --1- 172.21.15.64:0/1580161515 >> v1:172.21.15.100:6792/0 conn(0x7f52d009ca00 0x7f52d00a5600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6792/0 says I am v1:172.21.15.64:55054/0 (socket says 172.21.15.64:55054) 2024-09-17T15:50:01.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 -- 172.21.15.64:0/1580161515 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1834936366 0 0) 0x7f52d00ab640 con 0x7f52d009ca00 2024-09-17T15:50:01.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient(hunting): my global_id is 5596 2024-09-17T15:50:01.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 -- 172.21.15.64:0/1580161515 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52bc003520 con 0x7f52d009ca00 2024-09-17T15:50:01.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 -- 172.21.15.64:0/1580161515 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2815606422 0 0) 0x7f52b8001040 con 0x7f52d00a5d10 2024-09-17T15:50:01.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient(hunting): my global_id is 114852 2024-09-17T15:50:01.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 -- 172.21.15.64:0/1580161515 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52d00ab640 con 0x7f52d00a5d10 2024-09-17T15:50:01.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 -- 172.21.15.64:0/1580161515 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3655210036 0 0) 0x7f52d0136260 con 0x7f52d0135ba0 2024-09-17T15:50:01.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient(hunting): my global_id is 124241 2024-09-17T15:50:01.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 -- 172.21.15.64:0/1580161515 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52b8001040 con 0x7f52d0135ba0 2024-09-17T15:50:01.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 -- 172.21.15.64:0/1580161515 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3333254428 0 0) 0x7f52bc003520 con 0x7f52d009ca00 2024-09-17T15:50:01.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 -- 172.21.15.64:0/1580161515 >> v1:172.21.15.100:6791/0 conn(0x7f52d0135ba0 legacy=0x7f52d01415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 -- 172.21.15.64:0/1580161515 >> v1:172.21.15.100:6790/0 conn(0x7f52d00a5d10 legacy=0x7f52d013dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:50:01.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 monclient: found mon.noname-i 2024-09-17T15:50:01.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:01.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.100:6792/0 2024-09-17T15:50:01.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 -- 172.21.15.64:0/1580161515 --> v1:172.21.15.100:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d009cde0 con 0x7f52d009ca00 2024-09-17T15:50:01.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient: _finish_auth 0 2024-09-17T15:50:01.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:50:01.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:01.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 -- 172.21.15.64:0/1580161515 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f52c0001760 con 0x7f52d009ca00 2024-09-17T15:50:01.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-17T15:50:01.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient: dump: 2024-09-17T15:50:01.410 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.411 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52d5472640 5 monclient: authenticate success, global_id 5596 2024-09-17T15:50:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52d5472640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 -- 172.21.15.64:0/1580161515 <== mon.7 v1:172.21.15.100:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f52c0001980 con 0x7f52d009ca00 2024-09-17T15:50:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 1 -- 172.21.15.64:0/1580161515 <== mon.7 v1:172.21.15.100:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f52c0004b20 con 0x7f52d009ca00 2024-09-17T15:50:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52d5472640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52d5472640 4 set_mon_vals no callback set 2024-09-17T15:50:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:50:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.397+0000 7f52ce7fc640 10 monclient: dump: 2024-09-17T15:50:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.413 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.398+0000 7f52d5472640 10 monclient: shutdown 2024-09-17T15:50:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.398+0000 7f52d5472640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.398+0000 7f52d5472640 1 -- 172.21.15.64:0/1580161515 >> v1:172.21.15.100:6792/0 conn(0x7f52d009ca00 legacy=0x7f52d00a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.398+0000 7f52d5472640 1 -- 172.21.15.64:0/1580161515 shutdown_connections 2024-09-17T15:50:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.398+0000 7f52d5472640 1 -- 172.21.15.64:0/1580161515 >> 172.21.15.64:0/1580161515 conn(0x7f52d006dad0 msgr2=0x7f52d00ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.398+0000 7f52d5472640 1 -- 172.21.15.64:0/1580161515 shutdown_connections 2024-09-17T15:50:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.398+0000 7f52d5472640 1 -- 172.21.15.64:0/1580161515 wait complete. 2024-09-17T15:50:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.399+0000 7f52d5472640 10 monclient: build_initial_monmap 2024-09-17T15:50:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.399+0000 7f52d5472640 10 monclient: monmap: 2024-09-17T15:50:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:01.416 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.399+0000 7f52d5472640 1 Processor -- start 2024-09-17T15:50:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.399+0000 7f52d5472640 1 -- start start 2024-09-17T15:50:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.399+0000 7f52d5472640 10 monclient: init 2024-09-17T15:50:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.399+0000 7f52d5472640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.399+0000 7f52d5472640 10 monclient: _add_conns ranks=[0,7,4,2,6,8,1,3,5] 2024-09-17T15:50:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52d5472640 10 monclient(hunting): picked mon.noname-a con 0x7f52d009ca00 addr v1:172.21.15.64:6789/0 2024-09-17T15:50:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52d5472640 10 monclient(hunting): picked mon.noname-i con 0x7f52d00a5d10 addr v1:172.21.15.100:6792/0 2024-09-17T15:50:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52d5472640 10 monclient(hunting): picked mon.noname-c con 0x7f52d0135ba0 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:01.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52d5472640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52d00a6700 con 0x7f52d009ca00 2024-09-17T15:50:01.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52d5472640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52d00a6b70 con 0x7f52d0135ba0 2024-09-17T15:50:01.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52d5472640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52d012f6f0 con 0x7f52d00a5d10 2024-09-17T15:50:01.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52d5472640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:01.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52d5472640 10 monclient(hunting): authenticate will time out at 999.075116s 2024-09-17T15:50:01.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52ceffd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f52d009ca00 0x7f52d012e8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35194/0 (socket says 172.21.15.64:35194) 2024-09-17T15:50:01.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52cf7fe640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f52d0135ba0 0x7f52d01d5dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:52754/0 (socket says 172.21.15.64:52754) 2024-09-17T15:50:01.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52ceffd640 1 -- 172.21.15.64:0/3616159788 learned_addr learned my addr 172.21.15.64:0/3616159788 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:01.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3332164424 0 0) 0x7f52d00a6b70 con 0x7f52d0135ba0 2024-09-17T15:50:01.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): my global_id is 5791 2024-09-17T15:50:01.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52b0003520 con 0x7f52d0135ba0 2024-09-17T15:50:01.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2420100826 0 0) 0x7f52d012f6f0 con 0x7f52d00a5d10 2024-09-17T15:50:01.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): my global_id is 5605 2024-09-17T15:50:01.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52d00a6b70 con 0x7f52d00a5d10 2024-09-17T15:50:01.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3394962000 0 0) 0x7f52d009cde0 con 0x7f52d009ca00 2024-09-17T15:50:01.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): my global_id is 124146 2024-09-17T15:50:01.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52d012f6f0 con 0x7f52d009ca00 2024-09-17T15:50:01.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 460129595 0 0) 0x7f52d012f6f0 con 0x7f52d009ca00 2024-09-17T15:50:01.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52d009cde0 con 0x7f52d009ca00 2024-09-17T15:50:01.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 904559391 0 0) 0x7f52d00a6b70 con 0x7f52d00a5d10 2024-09-17T15:50:01.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52d012f6f0 con 0x7f52d00a5d10 2024-09-17T15:50:01.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1446689949 0 0) 0x7f52b0003520 con 0x7f52d0135ba0 2024-09-17T15:50:01.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52d00a6b70 con 0x7f52d0135ba0 2024-09-17T15:50:01.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f52c0001760 con 0x7f52d009ca00 2024-09-17T15:50:01.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): dump: 2024-09-17T15:50:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.420 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.421 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f52b8001080 con 0x7f52d00a5d10 2024-09-17T15:50:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:50:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): dump: 2024-09-17T15:50:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.422 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.422 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.422 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.422 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.422 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.422 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.422 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.422 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.422 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.422 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.422 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.422 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.422 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f52c80016e0 con 0x7f52d0135ba0 2024-09-17T15:50:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.400+0000 7f52c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 10 monclient(hunting): dump: 2024-09-17T15:50:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.423 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2051320704 0 0) 0x7f52d012f6f0 con 0x7f52d00a5d10 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 >> v1:172.21.15.64:6791/0 conn(0x7f52d0135ba0 legacy=0x7f52d01d5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 >> v1:172.21.15.64:6789/0 conn(0x7f52d009ca00 legacy=0x7f52d012e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 1 monclient: found mon.i 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:01.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:50:01.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 --> v1:172.21.15.100:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d0132300 con 0x7f52d00a5d10 2024-09-17T15:50:01.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 10 monclient: _finish_auth 0 2024-09-17T15:50:01.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:50:01.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:01.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52d5472640 5 monclient: authenticate success, global_id 5605 2024-09-17T15:50:01.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52d5472640 10 monclient: _renew_subs 2024-09-17T15:50:01.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52d5472640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:50:01.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52d5472640 1 -- 172.21.15.64:0/3616159788 --> v1:172.21.15.100:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f52d0130960 con 0x7f52d00a5d10 2024-09-17T15:50:01.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52d5472640 10 monclient: _renew_subs 2024-09-17T15:50:01.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52d5472640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:50:01.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52d5472640 1 -- 172.21.15.64:0/3616159788 --> v1:172.21.15.100:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f52d0132cc0 con 0x7f52d00a5d10 2024-09-17T15:50:01.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.7 v1:172.21.15.100:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f52b80047e0 con 0x7f52d00a5d10 2024-09-17T15:50:01.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.7 v1:172.21.15.100:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f52b8004cd0 con 0x7f52d00a5d10 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52cdffb640 4 set_mon_vals no callback set 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.401+0000 7f52c77fe640 10 monclient: dump: 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.426 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.427 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.427 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.427 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.427 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.427 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.427 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.427 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.402+0000 7f52d5472640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:01.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.402+0000 7f52d5472640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:01.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.402+0000 7f52d5472640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:50:01.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.402+0000 7f52d5472640 1 -- 172.21.15.64:0/3616159788 --> v1:172.21.15.100:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52b4005080 con 0x7f52d00a5d10 2024-09-17T15:50:01.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.402+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.7 v1:172.21.15.100:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f52b80115b0 con 0x7f52d00a5d10 2024-09-17T15:50:01.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.402+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.7 v1:172.21.15.100:6792/0 8 ==== osd_map(195..195 src has 1..195) ==== 3898+0+0 (unknown 636820965 0 0) 0x7f52b804e9f0 con 0x7f52d00a5d10 2024-09-17T15:50:01.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.402+0000 7f52ceffd640 10 monclient: get_auth_request con 0x7f52b0040cc0 auth_method 0 2024-09-17T15:50:01.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.406+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.7 v1:172.21.15.100:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f52d0135fe0 con 0x7f52d00a5d10 2024-09-17T15:50:01.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.406+0000 7f52c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:01.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.406+0000 7f52c77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:01.455 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:01.455 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38014]: starting. 2024-09-17T15:50:01.456 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38014]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-37917 2024-09-17T15:50:01.456 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38014]: shutting down. 2024-09-17T15:50:01.457 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:01.457 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:01.457 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:50:01.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:50:01.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.671+0000 7f52d5472640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:50:01.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.671+0000 7f52d5472640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:01.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.671+0000 7f52d5472640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:50:01.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.671+0000 7f52d5472640 1 -- 172.21.15.64:0/3616159788 --> v1:172.21.15.100:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f52b4005370 con 0x7f52d00a5d10 2024-09-17T15:50:01.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.672+0000 7f52c77fe640 1 -- 172.21.15.64:0/3616159788 <== mon.7 v1:172.21.15.100:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 3492669960) 0x7f52b80174c0 con 0x7f52d00a5d10 2024-09-17T15:50:01.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.672+0000 7f52c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:01.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.672+0000 7f52c77fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:01.674 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":24,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:50:01.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.674+0000 7f52d5472640 1 -- 172.21.15.64:0/3616159788 >> v1:172.21.15.100:6812/4210963484 conn(0x7f52b0040cc0 legacy=0x7f52b0043180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.674+0000 7f52d5472640 10 monclient: shutdown 2024-09-17T15:50:01.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.674+0000 7f52d5472640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:01.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.674+0000 7f52d5472640 1 -- 172.21.15.64:0/3616159788 >> v1:172.21.15.100:6792/0 conn(0x7f52d00a5d10 legacy=0x7f52d012efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.674+0000 7f52d5472640 1 -- 172.21.15.64:0/3616159788 shutdown_connections 2024-09-17T15:50:01.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.674+0000 7f52d5472640 1 -- 172.21.15.64:0/3616159788 >> 172.21.15.64:0/3616159788 conn(0x7f52d006dad0 msgr2=0x7f52d012c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:01.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.675+0000 7f52d5472640 1 -- 172.21.15.64:0/3616159788 shutdown_connections 2024-09-17T15:50:01.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.675+0000 7f52d5472640 1 -- 172.21.15.64:0/3616159788 wait complete. 2024-09-17T15:50:01.687 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-17T15:50:01.687 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:50:01.687 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-17T15:50:01.687 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:50:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.787+0000 7f0547631640 10 monclient: get_monmap_and_config 2024-09-17T15:50:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.787+0000 7f0547631640 10 monclient: build_initial_monmap 2024-09-17T15:50:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.787+0000 7f0547631640 10 monclient: monmap: 2024-09-17T15:50:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:01.788254+0000 2024-09-17T15:50:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:01.788254+0000 2024-09-17T15:50:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:01.791 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.787+0000 7f0547631640 1 Processor -- start 2024-09-17T15:50:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0547631640 1 -- start start 2024-09-17T15:50:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0547631640 10 monclient: init 2024-09-17T15:50:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0547631640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0547631640 10 monclient: _add_conns ranks=[7,2,4,8,1,0,3,6,5] 2024-09-17T15:50:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0547631640 10 monclient(hunting): picked mon.noname-i con 0x7f05400a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:50:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0547631640 10 monclient(hunting): picked mon.noname-b con 0x7f0540135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0547631640 10 monclient(hunting): picked mon.noname-c con 0x7f0540135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0547631640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f054009d6f0 con 0x7f0540135b50 2024-09-17T15:50:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0547631640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f054009d9d0 con 0x7f0540135f70 2024-09-17T15:50:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0547631640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f054009dcb0 con 0x7f05400a5520 2024-09-17T15:50:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0547631640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0547631640 10 monclient(hunting): authenticate will time out at 999.463754s 2024-09-17T15:50:01.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0544bcc640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f0540135b50 0x7f054013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46946/0 (socket says 172.21.15.64:46946) 2024-09-17T15:50:01.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0544bcc640 1 -- 172.21.15.64:0/3017020078 learned_addr learned my addr 172.21.15.64:0/3017020078 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:01.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.788+0000 7f0545bce640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f0540135f70 0x7f05401415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:52758/0 (socket says 172.21.15.64:52758) 2024-09-17T15:50:01.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 -- 172.21.15.64:0/3017020078 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129947824 0 0) 0x7f054009d9d0 con 0x7f0540135f70 2024-09-17T15:50:01.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient(hunting): my global_id is 5809 2024-09-17T15:50:01.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 -- 172.21.15.64:0/3017020078 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f052c003520 con 0x7f0540135f70 2024-09-17T15:50:01.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 -- 172.21.15.64:0/3017020078 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2499657570 0 0) 0x7f054009d6f0 con 0x7f0540135b50 2024-09-17T15:50:01.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient(hunting): my global_id is 124184 2024-09-17T15:50:01.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 -- 172.21.15.64:0/3017020078 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f054009d9d0 con 0x7f0540135b50 2024-09-17T15:50:01.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 -- 172.21.15.64:0/3017020078 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317400633 0 0) 0x7f054009dcb0 con 0x7f05400a5520 2024-09-17T15:50:01.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient(hunting): my global_id is 5623 2024-09-17T15:50:01.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 -- 172.21.15.64:0/3017020078 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f054009d6f0 con 0x7f05400a5520 2024-09-17T15:50:01.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 -- 172.21.15.64:0/3017020078 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4049461539 0 0) 0x7f052c003520 con 0x7f0540135f70 2024-09-17T15:50:01.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 -- 172.21.15.64:0/3017020078 >> v1:172.21.15.100:6792/0 conn(0x7f05400a5520 legacy=0x7f05400a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 -- 172.21.15.64:0/3017020078 >> v1:172.21.15.64:6790/0 conn(0x7f0540135b50 legacy=0x7f054013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient: _finish_hunting 0 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 monclient: found mon.noname-c 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 -- 172.21.15.64:0/3017020078 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054009ccd0 con 0x7f0540135f70 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient: _finish_auth 0 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient: _check_auth_tickets 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 -- 172.21.15.64:0/3017020078 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0538001bb0 con 0x7f0540135f70 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient: dump: 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.803 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.804 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.804 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.804 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.804 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.804 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.804 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.804 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.804 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.804 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.804 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.804 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.804 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.804 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0547631640 5 monclient: authenticate success, global_id 5809 2024-09-17T15:50:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0547631640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 -- 172.21.15.64:0/3017020078 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0538001d50 con 0x7f0540135f70 2024-09-17T15:50:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 1 -- 172.21.15.64:0/3017020078 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0538004ea0 con 0x7f0540135f70 2024-09-17T15:50:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0533fff640 10 monclient: dump: 2024-09-17T15:50:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.805 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0547631640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.789+0000 7f0547631640 4 set_mon_vals no callback set 2024-09-17T15:50:01.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.790+0000 7f0547631640 10 monclient: shutdown 2024-09-17T15:50:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.790+0000 7f0547631640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.790+0000 7f0547631640 1 -- 172.21.15.64:0/3017020078 >> v1:172.21.15.64:6791/0 conn(0x7f0540135f70 legacy=0x7f05401415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.790+0000 7f0547631640 1 -- 172.21.15.64:0/3017020078 shutdown_connections 2024-09-17T15:50:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.790+0000 7f0547631640 1 -- 172.21.15.64:0/3017020078 >> 172.21.15.64:0/3017020078 conn(0x7f054006db30 msgr2=0x7f05400ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.790+0000 7f0547631640 1 -- 172.21.15.64:0/3017020078 shutdown_connections 2024-09-17T15:50:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.790+0000 7f0547631640 1 -- 172.21.15.64:0/3017020078 wait complete. 2024-09-17T15:50:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.790+0000 7f0547631640 10 monclient: build_initial_monmap 2024-09-17T15:50:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.790+0000 7f0547631640 10 monclient: monmap: 2024-09-17T15:50:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:01.807 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:01.808 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0547631640 1 Processor -- start 2024-09-17T15:50:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0547631640 1 -- start start 2024-09-17T15:50:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0547631640 10 monclient: init 2024-09-17T15:50:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0547631640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0547631640 10 monclient: _add_conns ranks=[1,3,4,8,7,6,2,5,0] 2024-09-17T15:50:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0547631640 10 monclient(hunting): picked mon.noname-f con 0x7f05400a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0547631640 10 monclient(hunting): picked mon.noname-g con 0x7f0540135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0547631640 10 monclient(hunting): picked mon.noname-c con 0x7f0540135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0547631640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05400a8f70 con 0x7f05400a5520 2024-09-17T15:50:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0547631640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0540135310 con 0x7f0540135b50 2024-09-17T15:50:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0547631640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0540134e00 con 0x7f0540135f70 2024-09-17T15:50:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0547631640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0547631640 10 monclient(hunting): authenticate will time out at 999.466687s 2024-09-17T15:50:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0545bce640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f0540135f70 0x7f05401d5880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:52766/0 (socket says 172.21.15.64:52766) 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0545bce640 1 -- 172.21.15.64:0/1754144151 learned_addr learned my addr 172.21.15.64:0/1754144151 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2216984106 0 0) 0x7f0540134e00 con 0x7f0540135f70 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0531ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0531ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0531ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0531ffb640 10 monclient(hunting): my global_id is 5818 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.791+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0524003550 con 0x7f0540135f70 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048618356 0 0) 0x7f053c001a50 con 0x7f05400a5520 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): my global_id is 5905 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0540134e00 con 0x7f05400a5520 2024-09-17T15:50:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 329813225 0 0) 0x7f0540135310 con 0x7f0540135b50 2024-09-17T15:50:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): my global_id is 114861 2024-09-17T15:50:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f053c001a50 con 0x7f0540135b50 2024-09-17T15:50:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 100916993 0 0) 0x7f0524003550 con 0x7f0540135f70 2024-09-17T15:50:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0540135310 con 0x7f0540135f70 2024-09-17T15:50:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0538001b70 con 0x7f0540135f70 2024-09-17T15:50:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:50:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): dump: 2024-09-17T15:50:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 550983691 0 0) 0x7f0540134e00 con 0x7f05400a5520 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0524003550 con 0x7f05400a5520 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f053c001ce0 con 0x7f05400a5520 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): dump: 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.814 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 692202630 0 0) 0x7f053c001a50 con 0x7f0540135b50 2024-09-17T15:50:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0540134e00 con 0x7f0540135b50 2024-09-17T15:50:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0534001610 con 0x7f0540135b50 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient(hunting): dump: 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.815 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.816 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 555618278 0 0) 0x7f0540135310 con 0x7f0540135f70 2024-09-17T15:50:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 >> v1:172.21.15.100:6790/0 conn(0x7f0540135b50 legacy=0x7f05400a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 >> v1:172.21.15.100:6789/0 conn(0x7f05400a5520 legacy=0x7f05401cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:50:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 monclient: found mon.c 2024-09-17T15:50:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:50:01.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0540132fa0 con 0x7f0540135f70 2024-09-17T15:50:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient: _finish_auth 0 2024-09-17T15:50:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:50:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0547631640 5 monclient: authenticate success, global_id 5818 2024-09-17T15:50:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0547631640 10 monclient: _renew_subs 2024-09-17T15:50:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0547631640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:50:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0547631640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f05400a0100 con 0x7f0540135f70 2024-09-17T15:50:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0538004e90 con 0x7f0540135f70 2024-09-17T15:50:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0538005400 con 0x7f0540135f70 2024-09-17T15:50:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0531ffb640 10 monclient: dump: 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.818 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f05337fe640 4 set_mon_vals no callback set 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0547631640 10 monclient: _renew_subs 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0547631640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.792+0000 7f0547631640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f05400a5050 con 0x7f0540135f70 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.793+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f0538011d60 con 0x7f0540135f70 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.793+0000 7f0547631640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.793+0000 7f0547631640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f05400008d0 con 0x7f0540142f10 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(196..196 src has 1..196) ==== 3506+0+0 (unknown 4065408621 0 0) 0x7f053804ecd0 con 0x7f0540135f70 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 789852461 0 0) 0x7f05400008d0 con 0x7f0540142f10 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 10 monclient(hunting): my global_id is 124164 2024-09-17T15:50:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0540135310 con 0x7f0540142f10 2024-09-17T15:50:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0544bcc640 10 monclient: get_auth_request con 0x7f0524040cb0 auth_method 0 2024-09-17T15:50:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4202543582 0 0) 0x7f0540135310 con 0x7f0540142f10 2024-09-17T15:50:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05400008d0 con 0x7f0540142f10 2024-09-17T15:50:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f053c003c70 con 0x7f0540142f10 2024-09-17T15:50:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 10 monclient: dump: 2024-09-17T15:50:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.820 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.821 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2484762958 0 0) 0x7f05400008d0 con 0x7f0540142f10 2024-09-17T15:50:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.64:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05400a5330 con 0x7f0540142f10 2024-09-17T15:50:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f053c006480 con 0x7f0540142f10 2024-09-17T15:50:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 >> v1:172.21.15.64:6789/0 conn(0x7f0540142f10 legacy=0x7f054012d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.794+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 >> v1:172.21.15.64:6789/0 conn(0x7f0540142f10 legacy=0x7f054012d680 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:01.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.800+0000 7f0547631640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.800+0000 7f0547631640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f054002e830 con 0x7f054002e450 2024-09-17T15:50:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1719818020 0 0) 0x7f054002e830 con 0x7f054002e450 2024-09-17T15:50:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 10 monclient(hunting): my global_id is 124173 2024-09-17T15:50:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05400008d0 con 0x7f054002e450 2024-09-17T15:50:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4135080961 0 0) 0x7f05400008d0 con 0x7f054002e450 2024-09-17T15:50:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f054002e830 con 0x7f054002e450 2024-09-17T15:50:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f053c004680 con 0x7f054002e450 2024-09-17T15:50:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 10 monclient: dump: 2024-09-17T15:50:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.825 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.826 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.826 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.826 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.826 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.826 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3320583130 0 0) 0x7f054002e830 con 0x7f054002e450 2024-09-17T15:50:01.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 --> v1:172.21.15.64:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f054012e4a0 con 0x7f054002e450 2024-09-17T15:50:01.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 491655312) 0x7f053c006630 con 0x7f054002e450 2024-09-17T15:50:01.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:01.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:01.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 >> v1:172.21.15.64:6789/0 conn(0x7f054002e450 legacy=0x7f054012dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.801+0000 7f0531ffb640 1 -- 172.21.15.64:0/1754144151 >> v1:172.21.15.64:6789/0 conn(0x7f054002e450 legacy=0x7f054012dd90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:01.827 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:50:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:50:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 8719, 2024-09-17T15:50:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:50:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:50:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:50:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:50:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:50:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 3, 2024-09-17T15:50:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:01.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.802+0000 7f0547631640 1 -- 172.21.15.64:0/1754144151 >> v1:172.21.15.100:6812/4210963484 conn(0x7f0524040cb0 legacy=0x7f0524043170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.802+0000 7f0547631640 10 monclient: shutdown 2024-09-17T15:50:01.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.802+0000 7f0547631640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:01.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.802+0000 7f0547631640 1 -- 172.21.15.64:0/1754144151 >> v1:172.21.15.64:6791/0 conn(0x7f0540135f70 legacy=0x7f05401d5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.802+0000 7f0545bce640 1 -- 172.21.15.64:0/1754144151 reap_dead start 2024-09-17T15:50:01.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.802+0000 7f0545bce640 1 -- 172.21.15.64:0/1754144151 reap_dead start 2024-09-17T15:50:01.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.802+0000 7f0547631640 1 -- 172.21.15.64:0/1754144151 shutdown_connections 2024-09-17T15:50:01.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.802+0000 7f0547631640 1 -- 172.21.15.64:0/1754144151 >> 172.21.15.64:0/1754144151 conn(0x7f054006db30 msgr2=0x7f05401334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:01.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.802+0000 7f0547631640 1 -- 172.21.15.64:0/1754144151 shutdown_connections 2024-09-17T15:50:01.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.802+0000 7f0547631640 1 -- 172.21.15.64:0/1754144151 wait complete. 2024-09-17T15:50:01.848 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:50:01.848 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-17T15:50:01.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:50:01.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.949+0000 7fcc0f194640 10 monclient: get_monmap_and_config 2024-09-17T15:50:01.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.949+0000 7fcc0f194640 10 monclient: build_initial_monmap 2024-09-17T15:50:01.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.949+0000 7fcc0f194640 10 monclient: monmap: 2024-09-17T15:50:01.951 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:01.951 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.951 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:01.950471+0000 2024-09-17T15:50:01.951 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:01.950471+0000 2024-09-17T15:50:01.952 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:01.952 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.952 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:01.952 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:01.952 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:01.952 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:01.952 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:01.952 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:01.952 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:01.952 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:01.953 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:01.953 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.950+0000 7fcc0f194640 1 Processor -- start 2024-09-17T15:50:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.950+0000 7fcc0f194640 1 -- start start 2024-09-17T15:50:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.950+0000 7fcc0f194640 10 monclient: init 2024-09-17T15:50:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.950+0000 7fcc0f194640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.950+0000 7fcc0f194640 10 monclient: _add_conns ranks=[1,2,0,8,3,6,4,7,5] 2024-09-17T15:50:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.950+0000 7fcc0f194640 10 monclient(hunting): picked mon.noname-f con 0x7fcc080a54b0 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.950+0000 7fcc0f194640 10 monclient(hunting): picked mon.noname-b con 0x7fcc08135c80 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.950+0000 7fcc0f194640 10 monclient(hunting): picked mon.noname-a con 0x7fcc081360a0 addr v1:172.21.15.64:6789/0 2024-09-17T15:50:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.950+0000 7fcc0f194640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc080aacb0 con 0x7fcc081360a0 2024-09-17T15:50:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.950+0000 7fcc0f194640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc0809d780 con 0x7fcc080a54b0 2024-09-17T15:50:01.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.950+0000 7fcc0f194640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc0809da60 con 0x7fcc08135c80 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.950+0000 7fcc0f194640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.950+0000 7fcc0f194640 10 monclient(hunting): authenticate will time out at 999.625971s 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc07fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fcc08135c80 0x7fcc0813df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46960/0 (socket says 172.21.15.64:46960) 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc07fff640 1 -- 172.21.15.64:0/2768607794 learned_addr learned my addr 172.21.15.64:0/2768607794 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc0d731640 1 --1- 172.21.15.64:0/2768607794 >> v1:172.21.15.64:6789/0 conn(0x7fcc081360a0 0x7fcc08141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35218/0 (socket says 172.21.15.64:35218) 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 -- 172.21.15.64:0/2768607794 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2755182471 0 0) 0x7fcc080aacb0 con 0x7fcc081360a0 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient(hunting): my global_id is 124182 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 -- 172.21.15.64:0/2768607794 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbf8003520 con 0x7fcc081360a0 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 -- 172.21.15.64:0/2768607794 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2329296099 0 0) 0x7fcc0809da60 con 0x7fcc08135c80 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient(hunting): my global_id is 124193 2024-09-17T15:50:01.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 -- 172.21.15.64:0/2768607794 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc080aacb0 con 0x7fcc08135c80 2024-09-17T15:50:01.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 -- 172.21.15.64:0/2768607794 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3944839623 0 0) 0x7fcc0809d780 con 0x7fcc080a54b0 2024-09-17T15:50:01.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient(hunting): my global_id is 5914 2024-09-17T15:50:01.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 -- 172.21.15.64:0/2768607794 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc0809da60 con 0x7fcc080a54b0 2024-09-17T15:50:01.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 -- 172.21.15.64:0/2768607794 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2519432369 0 0) 0x7fcbf8003520 con 0x7fcc081360a0 2024-09-17T15:50:01.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 -- 172.21.15.64:0/2768607794 >> v1:172.21.15.64:6790/0 conn(0x7fcc08135c80 legacy=0x7fcc0813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 -- 172.21.15.64:0/2768607794 >> v1:172.21.15.100:6789/0 conn(0x7fcc080a54b0 legacy=0x7fcc080a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient: _finish_hunting 0 2024-09-17T15:50:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 monclient: found mon.noname-a 2024-09-17T15:50:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:50:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 -- 172.21.15.64:0/2768607794 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0809dd40 con 0x7fcc081360a0 2024-09-17T15:50:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient: _finish_auth 0 2024-09-17T15:50:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient: _check_auth_tickets 2024-09-17T15:50:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 -- 172.21.15.64:0/2768607794 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcc00001bb0 con 0x7fcc081360a0 2024-09-17T15:50:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc0f194640 5 monclient: authenticate success, global_id 124182 2024-09-17T15:50:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc0f194640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:01.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient: dump: 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.966 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc0f194640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 -- 172.21.15.64:0/2768607794 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc00001d50 con 0x7fcc081360a0 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 1 -- 172.21.15.64:0/2768607794 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcc00004ea0 con 0x7fcc081360a0 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc0f194640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc0f194640 4 set_mon_vals no callback set 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.951+0000 7fcc077fe640 10 monclient: dump: 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.967 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.968 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.968 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.968 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.968 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.968 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.968 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.968 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.968 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.968 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.968 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.968 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.968 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.968 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.952+0000 7fcc0f194640 10 monclient: shutdown 2024-09-17T15:50:01.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.952+0000 7fcc0f194640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:01.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.952+0000 7fcc0f194640 1 -- 172.21.15.64:0/2768607794 >> v1:172.21.15.64:6789/0 conn(0x7fcc081360a0 legacy=0x7fcc08141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.952+0000 7fcc0f194640 1 -- 172.21.15.64:0/2768607794 shutdown_connections 2024-09-17T15:50:01.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.952+0000 7fcc0f194640 1 -- 172.21.15.64:0/2768607794 >> 172.21.15.64:0/2768607794 conn(0x7fcc0806daa0 msgr2=0x7fcc080ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:01.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.952+0000 7fcc0f194640 1 -- 172.21.15.64:0/2768607794 shutdown_connections 2024-09-17T15:50:01.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.952+0000 7fcc0f194640 1 -- 172.21.15.64:0/2768607794 wait complete. 2024-09-17T15:50:01.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.952+0000 7fcc0f194640 10 monclient: build_initial_monmap 2024-09-17T15:50:01.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.952+0000 7fcc0f194640 10 monclient: monmap: 2024-09-17T15:50:01.970 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:01.970 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:01.970 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:01.970 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:01.970 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:01.972 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:01.972 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:01.972 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:01.972 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:01.972 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:01.972 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:01.973 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:01.973 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:01.973 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:01.973 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:01.973 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.953+0000 7fcc0f194640 1 Processor -- start 2024-09-17T15:50:01.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.953+0000 7fcc0f194640 1 -- start start 2024-09-17T15:50:01.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.953+0000 7fcc0f194640 10 monclient: init 2024-09-17T15:50:01.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.953+0000 7fcc0f194640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:01.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.953+0000 7fcc0f194640 10 monclient: _add_conns ranks=[0,6,2,1,4,5,3,7,8] 2024-09-17T15:50:01.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.953+0000 7fcc0f194640 10 monclient(hunting): picked mon.noname-a con 0x7fcc080a54b0 addr v1:172.21.15.64:6789/0 2024-09-17T15:50:01.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.953+0000 7fcc0f194640 10 monclient(hunting): picked mon.noname-d con 0x7fcc08135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:50:01.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.953+0000 7fcc0f194640 10 monclient(hunting): picked mon.noname-b con 0x7fcc081360a0 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:01.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc0f194640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc0809f950 con 0x7fcc080a54b0 2024-09-17T15:50:01.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc0f194640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc080a07c0 con 0x7fcc081360a0 2024-09-17T15:50:01.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc0cf30640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fcc080a54b0 0x7fcc08131a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35220/0 (socket says 172.21.15.64:35220) 2024-09-17T15:50:01.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc0f194640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc080a6970 con 0x7fcc08135c80 2024-09-17T15:50:01.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc0f194640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:01.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc0cf30640 1 -- 172.21.15.64:0/1515778869 learned_addr learned my addr 172.21.15.64:0/1515778869 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:01.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc0f194640 10 monclient(hunting): authenticate will time out at 999.629093s 2024-09-17T15:50:01.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3734792588 0 0) 0x7fcc0809f950 con 0x7fcc080a54b0 2024-09-17T15:50:01.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): my global_id is 124191 2024-09-17T15:50:01.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbec003520 con 0x7fcc080a54b0 2024-09-17T15:50:01.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1749929942 0 0) 0x7fcc080a6970 con 0x7fcc08135c80 2024-09-17T15:50:01.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): my global_id is 124206 2024-09-17T15:50:01.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc0809f950 con 0x7fcc08135c80 2024-09-17T15:50:01.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3636664976 0 0) 0x7fcc080a07c0 con 0x7fcc081360a0 2024-09-17T15:50:01.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): my global_id is 124202 2024-09-17T15:50:01.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc080a6970 con 0x7fcc081360a0 2024-09-17T15:50:01.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3905444114 0 0) 0x7fcbec003520 con 0x7fcc080a54b0 2024-09-17T15:50:01.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc080a07c0 con 0x7fcc080a54b0 2024-09-17T15:50:01.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcbfc001bb0 con 0x7fcc080a54b0 2024-09-17T15:50:01.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): dump: 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.978 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.979 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.979 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.979 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.979 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:01.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 235621737 0 0) 0x7fcc0809f950 con 0x7fcc08135c80 2024-09-17T15:50:01.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbec003520 con 0x7fcc08135c80 2024-09-17T15:50:01.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcbf4001430 con 0x7fcc08135c80 2024-09-17T15:50:01.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:01.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:50:01.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): dump: 2024-09-17T15:50:01.979 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.979 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.980 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.980 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.980 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.980 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.980 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.980 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.980 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.980 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.980 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.980 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.980 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.980 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.980 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.981 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 535402355 0 0) 0x7fcc080a6970 con 0x7fcc081360a0 2024-09-17T15:50:01.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc0809f950 con 0x7fcc081360a0 2024-09-17T15:50:01.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcc00001a70 con 0x7fcc081360a0 2024-09-17T15:50:01.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:01.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:01.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient(hunting): dump: 2024-09-17T15:50:01.981 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.981 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.982 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.982 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.982 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.982 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.982 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.982 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.982 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.982 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.982 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.982 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.982 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.982 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.982 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.983 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1202021075 0 0) 0x7fcc080a07c0 con 0x7fcc080a54b0 2024-09-17T15:50:01.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 >> v1:172.21.15.64:6792/0 conn(0x7fcc08135c80 legacy=0x7fcc08132110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 >> v1:172.21.15.64:6790/0 conn(0x7fcc081360a0 legacy=0x7fcc081d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient: _finish_hunting 0 2024-09-17T15:50:01.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 monclient: found mon.a 2024-09-17T15:50:01.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:01.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:50:01.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0809e460 con 0x7fcc080a54b0 2024-09-17T15:50:01.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient: _finish_auth 0 2024-09-17T15:50:01.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 10 monclient: _check_auth_tickets 2024-09-17T15:50:01.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.954+0000 7fcc057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:01.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc0f194640 5 monclient: authenticate success, global_id 124191 2024-09-17T15:50:01.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc0f194640 10 monclient: _renew_subs 2024-09-17T15:50:01.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc0f194640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:50:01.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc0f194640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc08132820 con 0x7fcc080a54b0 2024-09-17T15:50:01.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc0f194640 10 monclient: _renew_subs 2024-09-17T15:50:01.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc0f194640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:50:01.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc0f194640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc0809f260 con 0x7fcc080a54b0 2024-09-17T15:50:01.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcbfc0038c0 con 0x7fcc080a54b0 2024-09-17T15:50:01.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc057fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:01.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcbfc003e30 con 0x7fcc080a54b0 2024-09-17T15:50:01.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc06ffd640 4 set_mon_vals no callback set 2024-09-17T15:50:01.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:01.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc057fa640 10 monclient: dump: 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.985 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc0f194640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc0f194640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcbe80065c0 con 0x7fcbe8002860 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.955+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fcbfc003fd0 con 0x7fcc080a54b0 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(196..196 src has 1..196) ==== 3506+0+0 (unknown 4065408621 0 0) 0x7fcbfc04ea90 con 0x7fcc080a54b0 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1716813297 0 0) 0x7fcbe80065c0 con 0x7fcbe8002860 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 10 monclient(hunting): my global_id is 124211 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc080a07c0 con 0x7fcbe8002860 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 209766705 0 0) 0x7fcc080a07c0 con 0x7fcbe8002860 2024-09-17T15:50:01.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbe80065c0 con 0x7fcbe8002860 2024-09-17T15:50:01.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcbf4004b20 con 0x7fcbe8002860 2024-09-17T15:50:01.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:01.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 10 monclient: dump: 2024-09-17T15:50:01.987 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.987 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.987 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.987 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.987 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.988 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.988 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.988 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.988 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.988 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.988 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.988 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.988 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.988 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.989 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.990 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc0d731640 10 monclient: get_auth_request con 0x7fcbec040df0 auth_method 0 2024-09-17T15:50:01.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1523713885 0 0) 0x7fcbe80065c0 con 0x7fcbe8002860 2024-09-17T15:50:01.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.956+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcbe800a160 con 0x7fcbe8002860 2024-09-17T15:50:01.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.957+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcbf4007480 con 0x7fcbe8002860 2024-09-17T15:50:01.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.957+0000 7fcc057fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:01.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.957+0000 7fcc057fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:01.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.957+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 >> v1:172.21.15.64:6790/0 conn(0x7fcbe8002860 legacy=0x7fcbe8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.957+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 >> v1:172.21.15.64:6790/0 conn(0x7fcbe8002860 legacy=0x7fcbe8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:01.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc0f194640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:01.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc0f194640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcbe8009e00 con 0x7fcbe80090e0 2024-09-17T15:50:01.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4248348442 0 0) 0x7fcbe8009e00 con 0x7fcbe80090e0 2024-09-17T15:50:01.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:01.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 10 monclient(hunting): my global_id is 124220 2024-09-17T15:50:01.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbe80065c0 con 0x7fcbe80090e0 2024-09-17T15:50:01.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1159160985 0 0) 0x7fcbe80065c0 con 0x7fcbe80090e0 2024-09-17T15:50:01.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbe8009e00 con 0x7fcbe80090e0 2024-09-17T15:50:01.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcbf4005630 con 0x7fcbe80090e0 2024-09-17T15:50:01.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:01.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 10 monclient: dump: 2024-09-17T15:50:01.993 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:01.993 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:01.993 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.993 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:01.993 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.993 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:01.993 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:01.993 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:01.993 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:01.993 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:01.993 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:01.994 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:01.994 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:01.994 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:01.994 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:01.994 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:01.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4185769400 0 0) 0x7fcbe8009e00 con 0x7fcbe80090e0 2024-09-17T15:50:01.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.963+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 --> v1:172.21.15.64:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcbe800c3a0 con 0x7fcbe80090e0 2024-09-17T15:50:01.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 458376139) 0x7fcbf4006b60 con 0x7fcbe80090e0 2024-09-17T15:50:01.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc057fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:01.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc057fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:01.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 >> v1:172.21.15.64:6790/0 conn(0x7fcbe80090e0 legacy=0x7fcbe8009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc057fa640 1 -- 172.21.15.64:0/1515778869 >> v1:172.21.15.64:6790/0 conn(0x7fcbe80090e0 legacy=0x7fcbe8009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:01.995 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:50:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:50:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 8881, 2024-09-17T15:50:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:50:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:50:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:50:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:50:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:50:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 3, 2024-09-17T15:50:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:02.017 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:02.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc0f194640 1 -- 172.21.15.64:0/1515778869 >> v1:172.21.15.100:6812/4210963484 conn(0x7fcbec040df0 legacy=0x7fcbec0432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc0f194640 10 monclient: shutdown 2024-09-17T15:50:02.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc0f194640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc0f194640 1 -- 172.21.15.64:0/1515778869 >> v1:172.21.15.64:6789/0 conn(0x7fcc080a54b0 legacy=0x7fcc08131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc0d731640 1 -- 172.21.15.64:0/1515778869 reap_dead start 2024-09-17T15:50:02.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc0d731640 1 -- 172.21.15.64:0/1515778869 reap_dead start 2024-09-17T15:50:02.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc0f194640 1 -- 172.21.15.64:0/1515778869 shutdown_connections 2024-09-17T15:50:02.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc0f194640 1 -- 172.21.15.64:0/1515778869 >> 172.21.15.64:0/1515778869 conn(0x7fcc0806daa0 msgr2=0x7fcc080a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc0f194640 1 -- 172.21.15.64:0/1515778869 shutdown_connections 2024-09-17T15:50:02.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:01.964+0000 7fcc0f194640 1 -- 172.21.15.64:0/1515778869 wait complete. 2024-09-17T15:50:02.019 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:50:02.019 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-17T15:50:02.019 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:50:02.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.121+0000 7fea12b64640 10 monclient: get_monmap_and_config 2024-09-17T15:50:02.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.121+0000 7fea12b64640 10 monclient: build_initial_monmap 2024-09-17T15:50:02.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.121+0000 7fea12b64640 10 monclient: monmap: 2024-09-17T15:50:02.123 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:02.123 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.123 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:02.122442+0000 2024-09-17T15:50:02.123 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:02.122442+0000 2024-09-17T15:50:02.124 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:02.124 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.124 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:02.124 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:02.124 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:02.124 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:02.124 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:02.124 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:02.124 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:02.124 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:02.124 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:02.124 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.122+0000 7fea12b64640 1 Processor -- start 2024-09-17T15:50:02.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.122+0000 7fea12b64640 1 -- start start 2024-09-17T15:50:02.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.122+0000 7fea12b64640 10 monclient: init 2024-09-17T15:50:02.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.122+0000 7fea12b64640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.122+0000 7fea12b64640 10 monclient: _add_conns ranks=[5,4,3,6,1,8,0,7,2] 2024-09-17T15:50:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.122+0000 7fea12b64640 10 monclient(hunting): picked mon.noname-h con 0x7fea0c0a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:50:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.122+0000 7fea12b64640 10 monclient(hunting): picked mon.noname-c con 0x7fea0c135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.122+0000 7fea12b64640 10 monclient(hunting): picked mon.noname-g con 0x7fea0c135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.122+0000 7fea12b64640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea0c09d6f0 con 0x7fea0c135f70 2024-09-17T15:50:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.122+0000 7fea12b64640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea0c09d9d0 con 0x7fea0c135b50 2024-09-17T15:50:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.122+0000 7fea12b64640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea0c09dcb0 con 0x7fea0c0a5520 2024-09-17T15:50:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.122+0000 7fea12b64640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.122+0000 7fea12b64640 10 monclient(hunting): authenticate will time out at 999.797950s 2024-09-17T15:50:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0bfff640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fea0c135b50 0x7fea0c13deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:52772/0 (socket says 172.21.15.64:52772) 2024-09-17T15:50:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0bfff640 1 -- 172.21.15.64:0/182497661 learned_addr learned my addr 172.21.15.64:0/182497661 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 -- 172.21.15.64:0/182497661 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3608138177 0 0) 0x7fea0c09d9d0 con 0x7fea0c135b50 2024-09-17T15:50:02.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient(hunting): my global_id is 5827 2024-09-17T15:50:02.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 -- 172.21.15.64:0/182497661 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9fc003520 con 0x7fea0c135b50 2024-09-17T15:50:02.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 -- 172.21.15.64:0/182497661 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 684322277 0 0) 0x7fea0c09dcb0 con 0x7fea0c0a5520 2024-09-17T15:50:02.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient(hunting): my global_id is 124250 2024-09-17T15:50:02.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 -- 172.21.15.64:0/182497661 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea0c09d9d0 con 0x7fea0c0a5520 2024-09-17T15:50:02.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 -- 172.21.15.64:0/182497661 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3666592490 0 0) 0x7fea0c09d6f0 con 0x7fea0c135f70 2024-09-17T15:50:02.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient(hunting): my global_id is 114870 2024-09-17T15:50:02.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 -- 172.21.15.64:0/182497661 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea0c09dcb0 con 0x7fea0c135f70 2024-09-17T15:50:02.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 -- 172.21.15.64:0/182497661 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2007224002 0 0) 0x7fe9fc003520 con 0x7fea0c135b50 2024-09-17T15:50:02.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 -- 172.21.15.64:0/182497661 >> v1:172.21.15.100:6791/0 conn(0x7fea0c0a5520 legacy=0x7fea0c0a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 -- 172.21.15.64:0/182497661 >> v1:172.21.15.100:6790/0 conn(0x7fea0c135f70 legacy=0x7fea0c1415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:50:02.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 monclient: found mon.noname-c 2024-09-17T15:50:02.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:02.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:50:02.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 -- 172.21.15.64:0/182497661 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea0c09ccd0 con 0x7fea0c135b50 2024-09-17T15:50:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:50:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:50:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea12b64640 5 monclient: authenticate success, global_id 5827 2024-09-17T15:50:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 -- 172.21.15.64:0/182497661 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe9f8001830 con 0x7fea0c135b50 2024-09-17T15:50:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea12b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:50:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient: dump: 2024-09-17T15:50:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.136 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea12b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:02.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 -- 172.21.15.64:0/182497661 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe9f8001a50 con 0x7fea0c135b50 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 1 -- 172.21.15.64:0/182497661 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe9f8004be0 con 0x7fea0c135b50 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea0b7fe640 10 monclient: dump: 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.138 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.139 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.139 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.139 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.139 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.139 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.139 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.139 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea12b64640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:02.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.123+0000 7fea12b64640 4 set_mon_vals no callback set 2024-09-17T15:50:02.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.124+0000 7fea12b64640 10 monclient: shutdown 2024-09-17T15:50:02.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.124+0000 7fea12b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.124+0000 7fea12b64640 1 -- 172.21.15.64:0/182497661 >> v1:172.21.15.64:6791/0 conn(0x7fea0c135b50 legacy=0x7fea0c13deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.124+0000 7fea12b64640 1 -- 172.21.15.64:0/182497661 shutdown_connections 2024-09-17T15:50:02.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.124+0000 7fea12b64640 1 -- 172.21.15.64:0/182497661 >> 172.21.15.64:0/182497661 conn(0x7fea0c06db30 msgr2=0x7fea0c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.124+0000 7fea12b64640 1 -- 172.21.15.64:0/182497661 shutdown_connections 2024-09-17T15:50:02.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.124+0000 7fea12b64640 1 -- 172.21.15.64:0/182497661 wait complete. 2024-09-17T15:50:02.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.124+0000 7fea12b64640 10 monclient: build_initial_monmap 2024-09-17T15:50:02.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.124+0000 7fea12b64640 10 monclient: monmap: 2024-09-17T15:50:02.140 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:02.140 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:02.140 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:02.140 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:02.140 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:02.140 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:02.140 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:02.140 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:02.141 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:02.141 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:02.141 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:02.141 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:02.141 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:02.141 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:02.141 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:02.141 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.125+0000 7fea12b64640 1 Processor -- start 2024-09-17T15:50:02.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.125+0000 7fea12b64640 1 -- start start 2024-09-17T15:50:02.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.125+0000 7fea12b64640 10 monclient: init 2024-09-17T15:50:02.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.125+0000 7fea12b64640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:02.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.125+0000 7fea12b64640 10 monclient: _add_conns ranks=[7,5,3,1,0,8,2,6,4] 2024-09-17T15:50:02.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.125+0000 7fea12b64640 10 monclient(hunting): picked mon.noname-i con 0x7fea0c0a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:50:02.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.125+0000 7fea12b64640 10 monclient(hunting): picked mon.noname-h con 0x7fea0c135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:50:02.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.125+0000 7fea12b64640 10 monclient(hunting): picked mon.noname-g con 0x7fea0c135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:02.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.125+0000 7fea12b64640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea0c09f900 con 0x7fea0c135f70 2024-09-17T15:50:02.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.125+0000 7fea12b64640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea0c0a6420 con 0x7fea0c135b50 2024-09-17T15:50:02.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.125+0000 7fea12b64640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea0c134840 con 0x7fea0c0a5520 2024-09-17T15:50:02.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.125+0000 7fea12b64640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:02.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.125+0000 7fea12b64640 10 monclient(hunting): authenticate will time out at 999.800919s 2024-09-17T15:50:02.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea0bfff640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7fea0c135b50 0x7fea0c131f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:34514/0 (socket says 172.21.15.64:34514) 2024-09-17T15:50:02.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea0bfff640 1 -- 172.21.15.64:0/3067754076 learned_addr learned my addr 172.21.15.64:0/3067754076 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:02.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 540060911 0 0) 0x7fea0c0a6420 con 0x7fea0c135b50 2024-09-17T15:50:02.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): my global_id is 124259 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9f0003520 con 0x7fea0c135b50 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3579700792 0 0) 0x7fea0c09f900 con 0x7fea0c135f70 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): my global_id is 114879 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea0c0a6420 con 0x7fea0c135f70 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3028378835 0 0) 0x7fea0c134840 con 0x7fea0c0a5520 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): my global_id is 5632 2024-09-17T15:50:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea0c09f900 con 0x7fea0c0a5520 2024-09-17T15:50:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3010698000 0 0) 0x7fea0c0a6420 con 0x7fea0c135f70 2024-09-17T15:50:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fea0c134840 con 0x7fea0c135f70 2024-09-17T15:50:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2536152140 0 0) 0x7fe9f0003520 con 0x7fea0c135b50 2024-09-17T15:50:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fea0c0a6420 con 0x7fea0c135b50 2024-09-17T15:50:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fea04001cf0 con 0x7fea0c135f70 2024-09-17T15:50:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:50:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): dump: 2024-09-17T15:50:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:02.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe9f8004b20 con 0x7fea0c135b50 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): dump: 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.146 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.147 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.147 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.147 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.147 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.147 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2276121809 0 0) 0x7fea0c09f900 con 0x7fea0c0a5520 2024-09-17T15:50:02.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9f0003520 con 0x7fea0c0a5520 2024-09-17T15:50:02.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fea00001a50 con 0x7fea0c0a5520 2024-09-17T15:50:02.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:50:02.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient(hunting): dump: 2024-09-17T15:50:02.147 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.147 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.148 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.148 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.148 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.148 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.148 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.148 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.148 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.148 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.148 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.148 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.148 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.148 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.149 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.149 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 960371490 0 0) 0x7fea0c134840 con 0x7fea0c135f70 2024-09-17T15:50:02.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 >> v1:172.21.15.100:6792/0 conn(0x7fea0c0a5520 legacy=0x7fea0c131870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 >> v1:172.21.15.100:6791/0 conn(0x7fea0c135b50 legacy=0x7fea0c131f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient: _finish_hunting 0 2024-09-17T15:50:02.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 monclient: found mon.g 2024-09-17T15:50:02.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:02.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:50:02.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea0c0a0770 con 0x7fea0c135f70 2024-09-17T15:50:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient: _finish_auth 0 2024-09-17T15:50:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 10 monclient: _check_auth_tickets 2024-09-17T15:50:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea12b64640 5 monclient: authenticate success, global_id 114879 2024-09-17T15:50:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea12b64640 10 monclient: _renew_subs 2024-09-17T15:50:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea12b64640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:50:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.126+0000 7fea12b64640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fea0c09fed0 con 0x7fea0c135f70 2024-09-17T15:50:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.127+0000 7fea12b64640 10 monclient: _renew_subs 2024-09-17T15:50:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.127+0000 7fea12b64640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:50:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.127+0000 7fea12b64640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fea0c1d7160 con 0x7fea0c135f70 2024-09-17T15:50:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.127+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.3 v1:172.21.15.100:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fea04001fd0 con 0x7fea0c135f70 2024-09-17T15:50:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.127+0000 7fea097fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.127+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.3 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fea04003f80 con 0x7fea0c135f70 2024-09-17T15:50:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.127+0000 7fea097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.127+0000 7fea097fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:50:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.127+0000 7fea0affd640 4 set_mon_vals no callback set 2024-09-17T15:50:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.127+0000 7fea097fa640 10 monclient: dump: 2024-09-17T15:50:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.151 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.152 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.152 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.152 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.152 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.152 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.152 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.152 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.127+0000 7fea12b64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:02.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.127+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.3 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fea04010860 con 0x7fea0c135f70 2024-09-17T15:50:02.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.127+0000 7fea12b64640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe9ec0065c0 con 0x7fe9ec002860 2024-09-17T15:50:02.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.3 v1:172.21.15.100:6790/0 8 ==== osd_map(196..196 src has 1..196) ==== 3506+0+0 (unknown 4065408621 0 0) 0x7fea0404db70 con 0x7fea0c135f70 2024-09-17T15:50:02.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2549445175 0 0) 0x7fe9ec0065c0 con 0x7fe9ec002860 2024-09-17T15:50:02.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 10 monclient(hunting): my global_id is 5836 2024-09-17T15:50:02.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea0c134840 con 0x7fe9ec002860 2024-09-17T15:50:02.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea0bfff640 10 monclient: get_auth_request con 0x7fe9f0040ca0 auth_method 0 2024-09-17T15:50:02.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1416022752 0 0) 0x7fea0c134840 con 0x7fe9ec002860 2024-09-17T15:50:02.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9ec0065c0 con 0x7fe9ec002860 2024-09-17T15:50:02.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fea00003930 con 0x7fe9ec002860 2024-09-17T15:50:02.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:02.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 10 monclient: dump: 2024-09-17T15:50:02.153 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.154 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2418150218 0 0) 0x7fe9ec0065c0 con 0x7fe9ec002860 2024-09-17T15:50:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.128+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.64:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9ec00a160 con 0x7fe9ec002860 2024-09-17T15:50:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.129+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fea00003ac0 con 0x7fe9ec002860 2024-09-17T15:50:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.129+0000 7fea097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.129+0000 7fea097fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.129+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 >> v1:172.21.15.64:6791/0 conn(0x7fe9ec002860 legacy=0x7fe9ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.129+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 >> v1:172.21.15.64:6791/0 conn(0x7fe9ec002860 legacy=0x7fe9ec004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:02.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea12b64640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea12b64640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe9ec009e00 con 0x7fe9ec0090e0 2024-09-17T15:50:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3366389397 0 0) 0x7fe9ec009e00 con 0x7fe9ec0090e0 2024-09-17T15:50:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea097fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea097fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea097fa640 10 monclient(hunting): my global_id is 5845 2024-09-17T15:50:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9ec0065c0 con 0x7fe9ec0090e0 2024-09-17T15:50:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2088010934 0 0) 0x7fe9ec0065c0 con 0x7fe9ec0090e0 2024-09-17T15:50:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9ec009e00 con 0x7fe9ec0090e0 2024-09-17T15:50:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fea000042b0 con 0x7fe9ec0090e0 2024-09-17T15:50:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.135+0000 7fea097fa640 10 monclient: dump: 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.158 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.159 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.159 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2337367038 0 0) 0x7fe9ec009e00 con 0x7fe9ec0090e0 2024-09-17T15:50:02.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 --> v1:172.21.15.64:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe9ec00c3a0 con 0x7fe9ec0090e0 2024-09-17T15:50:02.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3403473925) 0x7fea00006570 con 0x7fe9ec0090e0 2024-09-17T15:50:02.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:02.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea097fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:02.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 >> v1:172.21.15.64:6791/0 conn(0x7fe9ec0090e0 legacy=0x7fe9ec009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea097fa640 1 -- 172.21.15.64:0/3067754076 >> v1:172.21.15.64:6791/0 conn(0x7fe9ec0090e0 legacy=0x7fe9ec009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:02.159 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:02.160 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:02.160 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:02.160 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:50:02.160 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:50:02.160 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 258793, 2024-09-17T15:50:02.160 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:02.160 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:02.160 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:02.160 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:02.160 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:02.160 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:50:02.160 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:50:02.160 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:50:02.161 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:50:02.161 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:50:02.161 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.161 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:50:02.161 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:02.161 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:02.161 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:02.161 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.161 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.161 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.161 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.161 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.161 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.162 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.163 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.163 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.163 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.163 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.163 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.163 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.163 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:02.163 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:02.163 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:02.163 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:02.163 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:02.164 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:02.164 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:02.164 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:02.164 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:02.164 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:02.164 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:02.164 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:02.164 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:02.164 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:02.164 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:02.164 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:02.164 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:02.164 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.165 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.165 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.165 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.165 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.165 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.165 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.165 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.165 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.165 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.165 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.165 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.165 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.167 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:02.168 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.168 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.168 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.168 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.168 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.168 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.168 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.168 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:02.169 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:02.169 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.169 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.169 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.169 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:02.169 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.169 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.169 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.169 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.169 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.169 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:02.171 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.172 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.173 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:02.176 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.176 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.176 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.176 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.176 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.176 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.176 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.177 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:02.178 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.179 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.179 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.179 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.179 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.179 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.179 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.179 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:02.179 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:02.179 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.181 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 4 2024-09-17T15:50:02.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.182 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:02.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.182 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.182 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.182 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:02.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.182 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.182 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:02.182 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:02.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea12b64640 1 -- 172.21.15.64:0/3067754076 >> v1:172.21.15.100:6812/4210963484 conn(0x7fe9f0040ca0 legacy=0x7fe9f0043160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea12b64640 10 monclient: shutdown 2024-09-17T15:50:02.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea12b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea12b64640 1 -- 172.21.15.64:0/3067754076 >> v1:172.21.15.100:6790/0 conn(0x7fea0c135f70 legacy=0x7fea0c132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea11101640 1 -- 172.21.15.64:0/3067754076 reap_dead start 2024-09-17T15:50:02.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea11101640 1 -- 172.21.15.64:0/3067754076 reap_dead start 2024-09-17T15:50:02.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea12b64640 1 -- 172.21.15.64:0/3067754076 shutdown_connections 2024-09-17T15:50:02.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea12b64640 1 -- 172.21.15.64:0/3067754076 >> 172.21.15.64:0/3067754076 conn(0x7fea0c06db30 msgr2=0x7fea0c0a76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea12b64640 1 -- 172.21.15.64:0/3067754076 shutdown_connections 2024-09-17T15:50:02.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.136+0000 7fea12b64640 1 -- 172.21.15.64:0/3067754076 wait complete. 2024-09-17T15:50:02.185 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:50:02.185 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-17T15:50:02.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-17T15:50:02.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.285+0000 7fb7ef1f5640 10 monclient: get_monmap_and_config 2024-09-17T15:50:02.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.285+0000 7fb7ef1f5640 10 monclient: build_initial_monmap 2024-09-17T15:50:02.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.285+0000 7fb7ef1f5640 10 monclient: monmap: 2024-09-17T15:50:02.287 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:02.287 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.287 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:02.286396+0000 2024-09-17T15:50:02.287 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:02.286396+0000 2024-09-17T15:50:02.287 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:02.287 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.288 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:02.288 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:02.288 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:02.288 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:02.288 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:02.288 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:02.288 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:02.288 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:02.288 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:02.288 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ef1f5640 1 Processor -- start 2024-09-17T15:50:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ef1f5640 1 -- start start 2024-09-17T15:50:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ef1f5640 10 monclient: init 2024-09-17T15:50:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ef1f5640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ef1f5640 10 monclient: _add_conns ranks=[2,1,3,6,0,8,7,5,4] 2024-09-17T15:50:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ef1f5640 10 monclient(hunting): picked mon.noname-b con 0x7fb7e80a54b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ef1f5640 10 monclient(hunting): picked mon.noname-f con 0x7fb7e8135c80 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ef1f5640 10 monclient(hunting): picked mon.noname-g con 0x7fb7e81360a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ef1f5640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e80aacb0 con 0x7fb7e8135c80 2024-09-17T15:50:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ef1f5640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e809d780 con 0x7fb7e80a54b0 2024-09-17T15:50:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ef1f5640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e809da60 con 0x7fb7e81360a0 2024-09-17T15:50:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ef1f5640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ef1f5640 10 monclient(hunting): authenticate will time out at 999.961858s 2024-09-17T15:50:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ecf91640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fb7e80a54b0 0x7fb7e80a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:47006/0 (socket says 172.21.15.64:47006) 2024-09-17T15:50:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.286+0000 7fb7ecf91640 1 -- 172.21.15.64:0/291112511 learned_addr learned my addr 172.21.15.64:0/291112511 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 -- 172.21.15.64:0/291112511 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2431232026 0 0) 0x7fb7e809d780 con 0x7fb7e80a54b0 2024-09-17T15:50:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient(hunting): my global_id is 124229 2024-09-17T15:50:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 -- 172.21.15.64:0/291112511 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7d8003520 con 0x7fb7e80a54b0 2024-09-17T15:50:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 -- 172.21.15.64:0/291112511 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1211944105 0 0) 0x7fb7e809da60 con 0x7fb7e81360a0 2024-09-17T15:50:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient(hunting): my global_id is 114888 2024-09-17T15:50:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 -- 172.21.15.64:0/291112511 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7e809d780 con 0x7fb7e81360a0 2024-09-17T15:50:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 -- 172.21.15.64:0/291112511 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3933878396 0 0) 0x7fb7e80aacb0 con 0x7fb7e8135c80 2024-09-17T15:50:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient(hunting): my global_id is 5923 2024-09-17T15:50:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 -- 172.21.15.64:0/291112511 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7e809da60 con 0x7fb7e8135c80 2024-09-17T15:50:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 -- 172.21.15.64:0/291112511 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 180454929 0 0) 0x7fb7d8003520 con 0x7fb7e80a54b0 2024-09-17T15:50:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 -- 172.21.15.64:0/291112511 >> v1:172.21.15.100:6790/0 conn(0x7fb7e81360a0 legacy=0x7fb7e8141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 -- 172.21.15.64:0/291112511 >> v1:172.21.15.100:6789/0 conn(0x7fb7e8135c80 legacy=0x7fb7e813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:50:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 monclient: found mon.noname-b 2024-09-17T15:50:02.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:02.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:50:02.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 -- 172.21.15.64:0/291112511 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e809dd40 con 0x7fb7e80a54b0 2024-09-17T15:50:02.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient: _finish_auth 0 2024-09-17T15:50:02.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:50:02.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:02.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 -- 172.21.15.64:0/291112511 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb7dc001830 con 0x7fb7e80a54b0 2024-09-17T15:50:02.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7ef1f5640 5 monclient: authenticate success, global_id 124229 2024-09-17T15:50:02.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7ef1f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:02.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient: dump: 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.301 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7ef1f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 -- 172.21.15.64:0/291112511 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb7dc0019d0 con 0x7fb7e80a54b0 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 1 -- 172.21.15.64:0/291112511 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb7dc004be0 con 0x7fb7e80a54b0 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7ef1f5640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7ef1f5640 4 set_mon_vals no callback set 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.287+0000 7fb7e77fe640 10 monclient: dump: 2024-09-17T15:50:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.303 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.304 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.288+0000 7fb7ef1f5640 10 monclient: shutdown 2024-09-17T15:50:02.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.288+0000 7fb7ef1f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.288+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291112511 >> v1:172.21.15.64:6790/0 conn(0x7fb7e80a54b0 legacy=0x7fb7e80a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.288+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291112511 shutdown_connections 2024-09-17T15:50:02.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.288+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291112511 >> 172.21.15.64:0/291112511 conn(0x7fb7e806daa0 msgr2=0x7fb7e80ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.288+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291112511 shutdown_connections 2024-09-17T15:50:02.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.288+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291112511 wait complete. 2024-09-17T15:50:02.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.288+0000 7fb7ef1f5640 10 monclient: build_initial_monmap 2024-09-17T15:50:02.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.288+0000 7fb7ef1f5640 10 monclient: monmap: 2024-09-17T15:50:02.304 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:02.304 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:02.304 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:02.305 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:02.305 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:02.305 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:02.305 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:02.305 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:02.305 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:02.305 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:02.305 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:02.305 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:02.305 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:02.305 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:02.305 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:02.305 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.288+0000 7fb7ef1f5640 1 Processor -- start 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ef1f5640 1 -- start start 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ef1f5640 10 monclient: init 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ef1f5640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ef1f5640 10 monclient: _add_conns ranks=[1,3,2,7,4,0,5,6,8] 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ef1f5640 10 monclient(hunting): picked mon.noname-f con 0x7fb7e80a54b0 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ef1f5640 10 monclient(hunting): picked mon.noname-g con 0x7fb7e8135c80 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ef1f5640 10 monclient(hunting): picked mon.noname-b con 0x7fb7e81360a0 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ef1f5640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e80a72b0 con 0x7fb7e80a54b0 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ef1f5640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e80a0320 con 0x7fb7e81360a0 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ef1f5640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e80a0600 con 0x7fb7e8135c80 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ef1f5640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ef1f5640 10 monclient(hunting): authenticate will time out at 999.964705s 2024-09-17T15:50:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ed792640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fb7e81360a0 0x7fb7e81d6240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:47014/0 (socket says 172.21.15.64:47014) 2024-09-17T15:50:02.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7ed792640 1 -- 172.21.15.64:0/291044645 learned_addr learned my addr 172.21.15.64:0/291044645 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:02.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.289+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1308571598 0 0) 0x7fb7e80a0320 con 0x7fb7e81360a0 2024-09-17T15:50:02.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): my global_id is 124238 2024-09-17T15:50:02.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7cc003550 con 0x7fb7e81360a0 2024-09-17T15:50:02.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4268934855 0 0) 0x7fb7e809dd40 con 0x7fb7e80a54b0 2024-09-17T15:50:02.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): my global_id is 5932 2024-09-17T15:50:02.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7e80a0320 con 0x7fb7e80a54b0 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1022667519 0 0) 0x7fb7e80a0600 con 0x7fb7e8135c80 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): my global_id is 114897 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7e809dd40 con 0x7fb7e8135c80 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3678481647 0 0) 0x7fb7cc003550 con 0x7fb7e81360a0 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7e80a0600 con 0x7fb7e81360a0 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb7e0001e30 con 0x7fb7e81360a0 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): dump: 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.308 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.309 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:02.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 761263466 0 0) 0x7fb7e80a0320 con 0x7fb7e80a54b0 2024-09-17T15:50:02.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7cc003550 con 0x7fb7e80a54b0 2024-09-17T15:50:02.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb7dc004b20 con 0x7fb7e80a54b0 2024-09-17T15:50:02.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:02.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient(hunting): dump: 2024-09-17T15:50:02.310 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.310 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.310 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.310 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.310 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.310 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.311 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.311 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.311 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.311 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.311 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.311 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.311 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.311 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.311 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.311 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3651489852 0 0) 0x7fb7e809dd40 con 0x7fb7e8135c80 2024-09-17T15:50:02.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7e80a0320 con 0x7fb7e8135c80 2024-09-17T15:50:02.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 94188638 0 0) 0x7fb7e80a0600 con 0x7fb7e81360a0 2024-09-17T15:50:02.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 >> v1:172.21.15.100:6790/0 conn(0x7fb7e8135c80 legacy=0x7fb7e809ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 >> v1:172.21.15.100:6789/0 conn(0x7fb7e80a54b0 legacy=0x7fb7e809e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient: _finish_hunting 0 2024-09-17T15:50:02.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 monclient: found mon.b 2024-09-17T15:50:02.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:02.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:02.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e809f120 con 0x7fb7e81360a0 2024-09-17T15:50:02.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient: _finish_auth 0 2024-09-17T15:50:02.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:50:02.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:02.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7ef1f5640 5 monclient: authenticate success, global_id 124238 2024-09-17T15:50:02.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7ef1f5640 10 monclient: _renew_subs 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7ef1f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e80a0bc0 con 0x7fb7e81360a0 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7ef1f5640 10 monclient: _renew_subs 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7ef1f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb7e80a4aa0 con 0x7fb7e81360a0 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb7e0003c40 con 0x7fb7e81360a0 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb7e0004170 con 0x7fb7e81360a0 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e6ffd640 4 set_mon_vals no callback set 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.290+0000 7fb7e57fa640 10 monclient: dump: 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.313 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.314 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.314 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.314 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.314 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.314 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.314 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.314 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.314 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.314 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.314 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.314 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.315 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.315 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.315 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.291+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fb7e0004310 con 0x7fb7e81360a0 2024-09-17T15:50:02.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.291+0000 7fb7ef1f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:02.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.291+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb7c80065c0 con 0x7fb7c8002860 2024-09-17T15:50:02.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.291+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(196..196 src has 1..196) ==== 3506+0+0 (unknown 4065408621 0 0) 0x7fb7e004ed70 con 0x7fb7e81360a0 2024-09-17T15:50:02.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e7fff640 10 monclient: get_auth_request con 0x7fb7cc040cd0 auth_method 0 2024-09-17T15:50:02.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2154740899 0 0) 0x7fb7c80065c0 con 0x7fb7c8002860 2024-09-17T15:50:02.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 10 monclient(hunting): my global_id is 124215 2024-09-17T15:50:02.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7e80a0600 con 0x7fb7c8002860 2024-09-17T15:50:02.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2621658625 0 0) 0x7fb7e80a0600 con 0x7fb7c8002860 2024-09-17T15:50:02.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7c80065c0 con 0x7fb7c8002860 2024-09-17T15:50:02.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb7dc0018e0 con 0x7fb7c8002860 2024-09-17T15:50:02.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:50:02.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 10 monclient: dump: 2024-09-17T15:50:02.316 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.316 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.316 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.316 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.316 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.316 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3266061275 0 0) 0x7fb7c80065c0 con 0x7fb7c8002860 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.64:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7c800a160 con 0x7fb7c8002860 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb7dc007480 con 0x7fb7c8002860 2024-09-17T15:50:02.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:02.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:02.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 >> v1:172.21.15.64:6792/0 conn(0x7fb7c8002860 legacy=0x7fb7c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.292+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 >> v1:172.21.15.64:6792/0 conn(0x7fb7c8002860 legacy=0x7fb7c8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:02.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.298+0000 7fb7ef1f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:02.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.298+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb7c8009e20 con 0x7fb7c80090e0 2024-09-17T15:50:02.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2891014752 0 0) 0x7fb7c8009e20 con 0x7fb7c80090e0 2024-09-17T15:50:02.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 10 monclient(hunting): my global_id is 124224 2024-09-17T15:50:02.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7c80065c0 con 0x7fb7c80090e0 2024-09-17T15:50:02.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 522121618 0 0) 0x7fb7c80065c0 con 0x7fb7c80090e0 2024-09-17T15:50:02.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7c8009e20 con 0x7fb7c80090e0 2024-09-17T15:50:02.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb7dc0055b0 con 0x7fb7c80090e0 2024-09-17T15:50:02.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:50:02.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 10 monclient: dump: 2024-09-17T15:50:02.320 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.320 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.321 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3877331553 0 0) 0x7fb7c8009e20 con 0x7fb7c80090e0 2024-09-17T15:50:02.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 --> v1:172.21.15.64:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb7c800c3f0 con 0x7fb7c80090e0 2024-09-17T15:50:02.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 1774318138) 0x7fb7dc007630 con 0x7fb7c80090e0 2024-09-17T15:50:02.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:02.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:02.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 >> v1:172.21.15.64:6792/0 conn(0x7fb7c80090e0 legacy=0x7fb7c8009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.299+0000 7fb7e57fa640 1 -- 172.21.15.64:0/291044645 >> v1:172.21.15.64:6792/0 conn(0x7fb7c80090e0 legacy=0x7fb7c8009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:02.322 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:50:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:50:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 9217, 2024-09-17T15:50:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:50:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:50:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:50:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.331 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:02.342 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:02.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.300+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291044645 >> v1:172.21.15.100:6812/4210963484 conn(0x7fb7cc040cd0 legacy=0x7fb7cc043190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.300+0000 7fb7ef1f5640 10 monclient: shutdown 2024-09-17T15:50:02.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.300+0000 7fb7ef1f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.300+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291044645 >> v1:172.21.15.64:6790/0 conn(0x7fb7e81360a0 legacy=0x7fb7e81d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.300+0000 7fb7ed792640 1 -- 172.21.15.64:0/291044645 reap_dead start 2024-09-17T15:50:02.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.300+0000 7fb7ed792640 1 -- 172.21.15.64:0/291044645 reap_dead start 2024-09-17T15:50:02.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.300+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291044645 shutdown_connections 2024-09-17T15:50:02.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.300+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291044645 >> 172.21.15.64:0/291044645 conn(0x7fb7e806daa0 msgr2=0x7fb7e8134000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.300+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291044645 shutdown_connections 2024-09-17T15:50:02.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.300+0000 7fb7ef1f5640 1 -- 172.21.15.64:0/291044645 wait complete. 2024-09-17T15:50:02.344 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:50:02.345 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-17T15:50:02.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-17T15:50:02.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.445+0000 7f83509d5640 10 monclient: get_monmap_and_config 2024-09-17T15:50:02.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.445+0000 7f83509d5640 10 monclient: build_initial_monmap 2024-09-17T15:50:02.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.445+0000 7f83509d5640 10 monclient: monmap: 2024-09-17T15:50:02.447 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:02.447 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.447 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:02.446300+0000 2024-09-17T15:50:02.447 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:02.446300+0000 2024-09-17T15:50:02.447 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:02.448 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.448 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:02.448 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:02.448 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:02.448 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:02.448 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:02.448 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:02.448 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:02.448 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:02.448 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:02.448 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.446+0000 7f83509d5640 1 Processor -- start 2024-09-17T15:50:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.446+0000 7f83509d5640 1 -- start start 2024-09-17T15:50:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.446+0000 7f83509d5640 10 monclient: init 2024-09-17T15:50:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.446+0000 7f83509d5640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.446+0000 7f83509d5640 10 monclient: _add_conns ranks=[3,2,4,0,6,5,8,1,7] 2024-09-17T15:50:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.446+0000 7f83509d5640 10 monclient(hunting): picked mon.noname-g con 0x7f834c0a5520 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.446+0000 7f83509d5640 10 monclient(hunting): picked mon.noname-b con 0x7f834c135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.446+0000 7f83509d5640 10 monclient(hunting): picked mon.noname-c con 0x7f834c135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.447+0000 7f83509d5640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f834c09d6f0 con 0x7f834c135b50 2024-09-17T15:50:02.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.447+0000 7f83509d5640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f834c09d9d0 con 0x7f834c0a5520 2024-09-17T15:50:02.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.447+0000 7f83509d5640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f834c09dcb0 con 0x7f834c135f70 2024-09-17T15:50:02.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.447+0000 7f83509d5640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:02.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.447+0000 7f83509d5640 10 monclient(hunting): authenticate will time out at 1000.122068s 2024-09-17T15:50:02.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.447+0000 7f834ad9d640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f834c135f70 0x7f834c1415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:52796/0 (socket says 172.21.15.64:52796) 2024-09-17T15:50:02.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.447+0000 7f8343fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f834c135b50 0x7f834c13deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:47016/0 (socket says 172.21.15.64:47016) 2024-09-17T15:50:02.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.447+0000 7f834ad9d640 1 -- 172.21.15.64:0/276198428 learned_addr learned my addr 172.21.15.64:0/276198428 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:02.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.447+0000 7f834a59c640 1 --1- 172.21.15.64:0/276198428 >> v1:172.21.15.100:6790/0 conn(0x7f834c0a5520 0x7f834c0a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:53366/0 (socket says 172.21.15.64:53366) 2024-09-17T15:50:02.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.449+0000 7f8349d9b640 1 -- 172.21.15.64:0/276198428 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1840871723 0 0) 0x7f834c09d6f0 con 0x7f834c135b50 2024-09-17T15:50:02.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.449+0000 7f8349d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.449+0000 7f8349d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.449+0000 7f8349d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.449+0000 7f8349d9b640 10 monclient(hunting): my global_id is 124247 2024-09-17T15:50:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.449+0000 7f8349d9b640 1 -- 172.21.15.64:0/276198428 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8338003520 con 0x7f834c135b50 2024-09-17T15:50:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.449+0000 7f8349d9b640 1 -- 172.21.15.64:0/276198428 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 654057703 0 0) 0x7f834c09d9d0 con 0x7f834c0a5520 2024-09-17T15:50:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.449+0000 7f8349d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.449+0000 7f8349d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.449+0000 7f8349d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.449+0000 7f8349d9b640 10 monclient(hunting): my global_id is 114906 2024-09-17T15:50:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.449+0000 7f8349d9b640 1 -- 172.21.15.64:0/276198428 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f834c09d6f0 con 0x7f834c0a5520 2024-09-17T15:50:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.449+0000 7f8349d9b640 1 -- 172.21.15.64:0/276198428 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4190762743 0 0) 0x7f8338003520 con 0x7f834c135b50 2024-09-17T15:50:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 1 -- 172.21.15.64:0/276198428 >> v1:172.21.15.64:6791/0 conn(0x7f834c135f70 legacy=0x7f834c1415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 1 -- 172.21.15.64:0/276198428 >> v1:172.21.15.100:6790/0 conn(0x7f834c0a5520 legacy=0x7f834c0a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:50:02.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 1 monclient: found mon.noname-b 2024-09-17T15:50:02.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:02.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:50:02.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 1 -- 172.21.15.64:0/276198428 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834c09ccd0 con 0x7f834c135b50 2024-09-17T15:50:02.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 10 monclient: _finish_auth 0 2024-09-17T15:50:02.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:50:02.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:02.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 1 -- 172.21.15.64:0/276198428 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8334001830 con 0x7f834c135b50 2024-09-17T15:50:02.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:50:02.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 10 monclient: dump: 2024-09-17T15:50:02.455 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.456 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.456 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.456 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.456 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.456 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.456 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.456 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.456 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.456 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.456 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.456 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.456 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.457 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.457 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.457 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f83509d5640 5 monclient: authenticate success, global_id 124247 2024-09-17T15:50:02.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f83509d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:02.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 1 -- 172.21.15.64:0/276198428 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8334001a90 con 0x7f834c135b50 2024-09-17T15:50:02.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:02.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f83509d5640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:02.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f83509d5640 4 set_mon_vals no callback set 2024-09-17T15:50:02.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 1 -- 172.21.15.64:0/276198428 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8334004b20 con 0x7f834c135b50 2024-09-17T15:50:02.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:02.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f8349d9b640 10 monclient: dump: 2024-09-17T15:50:02.458 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.458 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.458 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.458 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.458 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.458 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.458 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.458 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.458 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.458 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.458 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.458 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.459 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.459 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.459 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.459 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.450+0000 7f83509d5640 10 monclient: shutdown 2024-09-17T15:50:02.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.451+0000 7f83509d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.451+0000 7f83509d5640 1 -- 172.21.15.64:0/276198428 >> v1:172.21.15.64:6790/0 conn(0x7f834c135b50 legacy=0x7f834c13deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.451+0000 7f83509d5640 1 -- 172.21.15.64:0/276198428 shutdown_connections 2024-09-17T15:50:02.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.451+0000 7f83509d5640 1 -- 172.21.15.64:0/276198428 >> 172.21.15.64:0/276198428 conn(0x7f834c06db30 msgr2=0x7f834c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.451+0000 7f83509d5640 1 -- 172.21.15.64:0/276198428 shutdown_connections 2024-09-17T15:50:02.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.451+0000 7f83509d5640 1 -- 172.21.15.64:0/276198428 wait complete. 2024-09-17T15:50:02.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.451+0000 7f83509d5640 10 monclient: build_initial_monmap 2024-09-17T15:50:02.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.451+0000 7f83509d5640 10 monclient: monmap: 2024-09-17T15:50:02.460 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:02.460 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:02.460 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:02.460 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:02.460 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:02.461 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:02.461 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:02.461 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:02.461 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:02.461 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:02.461 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:02.461 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:02.461 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:02.461 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:02.461 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:02.461 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f83509d5640 1 Processor -- start 2024-09-17T15:50:02.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f83509d5640 1 -- start start 2024-09-17T15:50:02.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f83509d5640 10 monclient: init 2024-09-17T15:50:02.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f83509d5640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:02.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f83509d5640 10 monclient: _add_conns ranks=[2,8,3,4,0,7,1,6,5] 2024-09-17T15:50:02.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f83509d5640 10 monclient(hunting): picked mon.noname-b con 0x7f834c0a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:02.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f83509d5640 10 monclient(hunting): picked mon.noname-e con 0x7f834c135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:50:02.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f83509d5640 10 monclient(hunting): picked mon.noname-g con 0x7f834c135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:02.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f83509d5640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f834c0a0790 con 0x7f834c0a5520 2024-09-17T15:50:02.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f83509d5640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f834c134f50 con 0x7f834c135f70 2024-09-17T15:50:02.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f83509d5640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f834c0a9120 con 0x7f834c135b50 2024-09-17T15:50:02.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f83509d5640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:02.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f83509d5640 10 monclient(hunting): authenticate will time out at 1000.127894s 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f834a59c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f834c0a5520 0x7f834c131920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:47030/0 (socket says 172.21.15.64:47030) 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f834a59c640 1 -- 172.21.15.64:0/1288978047 learned_addr learned my addr 172.21.15.64:0/1288978047 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.452+0000 7f8343fff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f834c135b50 0x7f834c132030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:58132/0 (socket says 172.21.15.64:58132) 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3382161597 0 0) 0x7f834c134f50 con 0x7f834c135f70 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 10 monclient(hunting): my global_id is 114915 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f832c003550 con 0x7f834c135f70 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1147410311 0 0) 0x7f832c003550 con 0x7f834c135f70 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f834c134f50 con 0x7f834c135f70 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8344001ca0 con 0x7f834c135f70 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:50:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 10 monclient(hunting): dump: 2024-09-17T15:50:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.467 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1417244062 0 0) 0x7f834c134f50 con 0x7f834c135f70 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 >> v1:172.21.15.64:6793/0 conn(0x7f834c135b50 legacy=0x7f834c132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 >> v1:172.21.15.64:6790/0 conn(0x7f834c0a5520 legacy=0x7f834c131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 1 monclient: found mon.g 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834c135310 con 0x7f834c135f70 2024-09-17T15:50:02.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.456+0000 7f8342ffd640 10 monclient: _finish_auth 0 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f8342ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f8342ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f83509d5640 5 monclient: authenticate success, global_id 114915 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f83509d5640 10 monclient: _renew_subs 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f83509d5640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f83509d5640 1 -- 172.21.15.64:0/1288978047 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f834c0a51a0 con 0x7f834c135f70 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f83509d5640 10 monclient: _renew_subs 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f83509d5640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f83509d5640 1 -- 172.21.15.64:0/1288978047 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f834c132740 con 0x7f834c135f70 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.3 v1:172.21.15.100:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8344004b60 con 0x7f834c135f70 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f8342ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.3 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8344005110 con 0x7f834c135f70 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f8342ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f8342ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:50:02.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f8342ffd640 10 monclient: dump: 2024-09-17T15:50:02.470 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.470 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.470 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.470 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.470 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.470 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.470 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.470 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.470 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.470 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.470 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.470 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.470 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.471 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.471 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.471 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.457+0000 7f834959a640 4 set_mon_vals no callback set 2024-09-17T15:50:02.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.458+0000 7f83509d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:02.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.458+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.3 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f83440052b0 con 0x7f834c135f70 2024-09-17T15:50:02.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.458+0000 7f83509d5640 1 -- 172.21.15.64:0/1288978047 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f83300065c0 con 0x7f8330002860 2024-09-17T15:50:02.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.459+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.3 v1:172.21.15.100:6790/0 8 ==== osd_map(197..197 src has 1..197) ==== 3898+0+0 (unknown 1479433347 0 0) 0x7f834404ef00 con 0x7f834c135f70 2024-09-17T15:50:02.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.459+0000 7f8343fff640 10 monclient: get_auth_request con 0x7f832c0402f0 auth_method 0 2024-09-17T15:50:02.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.468+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2533864414 0 0) 0x7f83300065c0 con 0x7f8330002860 2024-09-17T15:50:02.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.468+0000 7f8342ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.468+0000 7f8342ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.468+0000 7f8342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.468+0000 7f8342ffd640 10 monclient(hunting): my global_id is 5777 2024-09-17T15:50:02.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.468+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f834c134f50 con 0x7f8330002860 2024-09-17T15:50:02.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.468+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1360191817 0 0) 0x7f834c134f50 con 0x7f8330002860 2024-09-17T15:50:02.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.468+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83300065c0 con 0x7f8330002860 2024-09-17T15:50:02.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.469+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f833c001480 con 0x7f8330002860 2024-09-17T15:50:02.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.469+0000 7f8342ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.469+0000 7f8342ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:50:02.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.469+0000 7f8342ffd640 10 monclient: dump: 2024-09-17T15:50:02.473 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.473 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.473 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.473 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.469+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3614744734 0 0) 0x7f83300065c0 con 0x7f8330002860 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.469+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 --> v1:172.21.15.64:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f833000a160 con 0x7f8330002860 2024-09-17T15:50:02.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.469+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f833c0016a0 con 0x7f8330002860 2024-09-17T15:50:02.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.469+0000 7f8342ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:02.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.469+0000 7f8342ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:02.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.469+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 >> v1:172.21.15.64:6793/0 conn(0x7f8330002860 legacy=0x7f8330004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.469+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 >> v1:172.21.15.64:6793/0 conn(0x7f8330002860 legacy=0x7f8330004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:02.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f83509d5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:02.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f83509d5640 1 -- 172.21.15.64:0/1288978047 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8330009e70 con 0x7f8330009130 2024-09-17T15:50:02.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 80653829 0 0) 0x7f8330009e70 con 0x7f8330009130 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 10 monclient(hunting): my global_id is 5786 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83300065c0 con 0x7f8330009130 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 182812094 0 0) 0x7f83300065c0 con 0x7f8330009130 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8330009e70 con 0x7f8330009130 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f833c004d10 con 0x7f8330009130 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 10 monclient: dump: 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.482 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.483 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.483 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.483 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.483 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.483 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.483 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.483 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.483 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.483 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.484 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.484 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.484 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.484 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1871547147 0 0) 0x7f8330009e70 con 0x7f8330009130 2024-09-17T15:50:02.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.479+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 --> v1:172.21.15.64:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f833000c410 con 0x7f8330009130 2024-09-17T15:50:02.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 602353045) 0x7f833c006e30 con 0x7f8330009130 2024-09-17T15:50:02.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f8342ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:02.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f8342ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:02.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 >> v1:172.21.15.64:6793/0 conn(0x7f8330009130 legacy=0x7f83300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f8342ffd640 1 -- 172.21.15.64:0/1288978047 >> v1:172.21.15.64:6793/0 conn(0x7f8330009130 legacy=0x7f83300095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 259126, 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:50:02.485 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:50:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:50:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:50:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:50:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.486 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.487 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:02.488 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.489 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.490 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.491 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.493 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.495 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.496 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.497 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:02.498 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.499 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.500 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.501 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.502 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.503 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:50:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.504 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:50:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:50:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.505 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:02.505 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:02.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f83509d5640 1 -- 172.21.15.64:0/1288978047 >> v1:172.21.15.100:6812/4210963484 conn(0x7f832c0402f0 legacy=0x7f832c0427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f83509d5640 10 monclient: shutdown 2024-09-17T15:50:02.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f83509d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f83509d5640 1 -- 172.21.15.64:0/1288978047 >> v1:172.21.15.100:6790/0 conn(0x7f834c135f70 legacy=0x7f834c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f834ad9d640 1 -- 172.21.15.64:0/1288978047 reap_dead start 2024-09-17T15:50:02.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f834ad9d640 1 -- 172.21.15.64:0/1288978047 reap_dead start 2024-09-17T15:50:02.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f83509d5640 1 -- 172.21.15.64:0/1288978047 shutdown_connections 2024-09-17T15:50:02.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f83509d5640 1 -- 172.21.15.64:0/1288978047 >> 172.21.15.64:0/1288978047 conn(0x7f834c06db30 msgr2=0x7f834c0a75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f83509d5640 1 -- 172.21.15.64:0/1288978047 shutdown_connections 2024-09-17T15:50:02.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.480+0000 7f83509d5640 1 -- 172.21.15.64:0/1288978047 wait complete. 2024-09-17T15:50:02.508 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:50:02.508 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-17T15:50:02.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-17T15:50:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.608+0000 7f4a5a953640 10 monclient: get_monmap_and_config 2024-09-17T15:50:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.608+0000 7f4a5a953640 10 monclient: build_initial_monmap 2024-09-17T15:50:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.608+0000 7f4a5a953640 10 monclient: monmap: 2024-09-17T15:50:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:02.609026+0000 2024-09-17T15:50:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:02.609026+0000 2024-09-17T15:50:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:02.610 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:02.611 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:02.611 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:02.611 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:02.611 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:02.611 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:02.611 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:02.611 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.608+0000 7f4a5a953640 1 Processor -- start 2024-09-17T15:50:02.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.608+0000 7f4a5a953640 1 -- start start 2024-09-17T15:50:02.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.608+0000 7f4a5a953640 10 monclient: init 2024-09-17T15:50:02.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.609+0000 7f4a5a953640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:02.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.609+0000 7f4a5a953640 10 monclient: _add_conns ranks=[5,7,1,3,6,8,4,0,2] 2024-09-17T15:50:02.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.609+0000 7f4a5a953640 10 monclient(hunting): picked mon.noname-h con 0x7f4a540a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:50:02.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.609+0000 7f4a5a953640 10 monclient(hunting): picked mon.noname-i con 0x7f4a54135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:50:02.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.609+0000 7f4a5a953640 10 monclient(hunting): picked mon.noname-f con 0x7f4a54135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:02.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.609+0000 7f4a5a953640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a5409d6f0 con 0x7f4a54135f70 2024-09-17T15:50:02.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.609+0000 7f4a5a953640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a5409d9d0 con 0x7f4a540a5520 2024-09-17T15:50:02.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.609+0000 7f4a5a953640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a5409dcb0 con 0x7f4a54135b50 2024-09-17T15:50:02.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.609+0000 7f4a5a953640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:02.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.609+0000 7f4a5a953640 10 monclient(hunting): authenticate will time out at 1000.284529s 2024-09-17T15:50:02.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.609+0000 7f4a58ef0640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f4a54135f70 0x7f4a541415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.64:35688/0 (socket says 172.21.15.64:35688) 2024-09-17T15:50:02.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.609+0000 7f4a58ef0640 1 -- 172.21.15.64:0/2354720401 learned_addr learned my addr 172.21.15.64:0/2354720401 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:02.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.609+0000 7f4a53fff640 1 --1- 172.21.15.64:0/2354720401 >> v1:172.21.15.100:6791/0 conn(0x7f4a540a5520 0x7f4a540a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:34530/0 (socket says 172.21.15.64:34530) 2024-09-17T15:50:02.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 -- 172.21.15.64:0/2354720401 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2607853025 0 0) 0x7f4a5409d6f0 con 0x7f4a54135f70 2024-09-17T15:50:02.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient(hunting): my global_id is 5941 2024-09-17T15:50:02.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 -- 172.21.15.64:0/2354720401 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a3c003520 con 0x7f4a54135f70 2024-09-17T15:50:02.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 -- 172.21.15.64:0/2354720401 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2325830248 0 0) 0x7f4a5409dcb0 con 0x7f4a54135b50 2024-09-17T15:50:02.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient(hunting): my global_id is 5641 2024-09-17T15:50:02.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 -- 172.21.15.64:0/2354720401 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a5409d6f0 con 0x7f4a54135b50 2024-09-17T15:50:02.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 -- 172.21.15.64:0/2354720401 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2072499937 0 0) 0x7f4a5409d9d0 con 0x7f4a540a5520 2024-09-17T15:50:02.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient(hunting): my global_id is 124268 2024-09-17T15:50:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 -- 172.21.15.64:0/2354720401 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a5409dcb0 con 0x7f4a540a5520 2024-09-17T15:50:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 -- 172.21.15.64:0/2354720401 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2029478145 0 0) 0x7f4a3c003520 con 0x7f4a54135f70 2024-09-17T15:50:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 -- 172.21.15.64:0/2354720401 >> v1:172.21.15.100:6792/0 conn(0x7f4a54135b50 legacy=0x7f4a5413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 -- 172.21.15.64:0/2354720401 >> v1:172.21.15.100:6791/0 conn(0x7f4a540a5520 legacy=0x7f4a540a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:50:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 monclient: found mon.noname-f 2024-09-17T15:50:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.100:6789/0 2024-09-17T15:50:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 -- 172.21.15.64:0/2354720401 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5409ccd0 con 0x7f4a54135f70 2024-09-17T15:50:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient: _finish_auth 0 2024-09-17T15:50:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:50:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:02.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 -- 172.21.15.64:0/2354720401 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a4c001ae0 con 0x7f4a54135f70 2024-09-17T15:50:02.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a5a953640 5 monclient: authenticate success, global_id 5941 2024-09-17T15:50:02.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a5a953640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:02.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:50:02.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient: dump: 2024-09-17T15:50:02.624 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.624 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.624 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.624 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.624 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.624 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.624 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.624 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.625 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.625 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.625 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.625 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.625 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.625 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.625 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.625 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a5a953640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:02.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 -- 172.21.15.64:0/2354720401 <== mon.1 v1:172.21.15.100:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4a4c001c80 con 0x7f4a54135f70 2024-09-17T15:50:02.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:02.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a5a953640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:02.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 1 -- 172.21.15.64:0/2354720401 <== mon.1 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a4c004f60 con 0x7f4a54135f70 2024-09-17T15:50:02.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a5a953640 4 set_mon_vals no callback set 2024-09-17T15:50:02.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:02.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a52ffd640 10 monclient: dump: 2024-09-17T15:50:02.626 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.626 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.626 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.626 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.626 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.626 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.626 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.626 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.626 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.627 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.627 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.627 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.627 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.627 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.627 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.627 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a5a953640 10 monclient: shutdown 2024-09-17T15:50:02.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a5a953640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a5a953640 1 -- 172.21.15.64:0/2354720401 >> v1:172.21.15.100:6789/0 conn(0x7f4a54135f70 legacy=0x7f4a541415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a5a953640 1 -- 172.21.15.64:0/2354720401 shutdown_connections 2024-09-17T15:50:02.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.610+0000 7f4a5a953640 1 -- 172.21.15.64:0/2354720401 >> 172.21.15.64:0/2354720401 conn(0x7f4a5406db30 msgr2=0x7f4a540ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.611+0000 7f4a5a953640 1 -- 172.21.15.64:0/2354720401 shutdown_connections 2024-09-17T15:50:02.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.611+0000 7f4a5a953640 1 -- 172.21.15.64:0/2354720401 wait complete. 2024-09-17T15:50:02.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.611+0000 7f4a5a953640 10 monclient: build_initial_monmap 2024-09-17T15:50:02.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.611+0000 7f4a5a953640 10 monclient: monmap: 2024-09-17T15:50:02.628 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:02.628 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:02.628 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:02.628 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:02.628 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:02.628 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:02.628 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:02.629 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:02.629 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:02.629 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:02.629 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:02.629 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:02.629 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:02.629 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:02.629 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:02.629 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.611+0000 7f4a5a953640 1 Processor -- start 2024-09-17T15:50:02.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a5a953640 1 -- start start 2024-09-17T15:50:02.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a5a953640 10 monclient: init 2024-09-17T15:50:02.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a5a953640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:02.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a5a953640 10 monclient: _add_conns ranks=[4,2,7,6,0,3,1,5,8] 2024-09-17T15:50:02.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a5a953640 10 monclient(hunting): picked mon.noname-c con 0x7f4a540a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:02.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a5a953640 10 monclient(hunting): picked mon.noname-b con 0x7f4a54135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:02.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a5a953640 10 monclient(hunting): picked mon.noname-i con 0x7f4a54135f70 addr v1:172.21.15.100:6792/0 2024-09-17T15:50:02.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a5a953640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a540a0390 con 0x7f4a54135b50 2024-09-17T15:50:02.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a5a953640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a5409f0a0 con 0x7f4a540a5520 2024-09-17T15:50:02.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a5a953640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a54132680 con 0x7f4a54135f70 2024-09-17T15:50:02.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a5a953640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:02.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a5a953640 10 monclient(hunting): authenticate will time out at 1000.287571s 2024-09-17T15:50:02.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a53fff640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f4a540a5520 0x7f4a54131860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:52808/0 (socket says 172.21.15.64:52808) 2024-09-17T15:50:02.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a53fff640 1 -- 172.21.15.64:0/2024525200 learned_addr learned my addr 172.21.15.64:0/2024525200 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:02.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 518319055 0 0) 0x7f4a5409f0a0 con 0x7f4a540a5520 2024-09-17T15:50:02.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a50ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a50ff9640 10 monclient(hunting): my global_id is 5854 2024-09-17T15:50:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a40003520 con 0x7f4a540a5520 2024-09-17T15:50:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4052460659 0 0) 0x7f4a540a0390 con 0x7f4a54135b50 2024-09-17T15:50:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a50ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a50ff9640 10 monclient(hunting): my global_id is 124256 2024-09-17T15:50:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a5409f0a0 con 0x7f4a54135b50 2024-09-17T15:50:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.612+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2401605784 0 0) 0x7f4a54132680 con 0x7f4a54135f70 2024-09-17T15:50:02.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): my global_id is 5650 2024-09-17T15:50:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a540a0390 con 0x7f4a54135f70 2024-09-17T15:50:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 281457100 0 0) 0x7f4a40003520 con 0x7f4a540a5520 2024-09-17T15:50:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a54132680 con 0x7f4a540a5520 2024-09-17T15:50:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a44001c60 con 0x7f4a540a5520 2024-09-17T15:50:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:50:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): dump: 2024-09-17T15:50:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.632 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.633 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.633 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.633 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.633 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.633 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.633 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.633 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.633 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.633 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.633 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.633 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.633 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.633 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3140450538 0 0) 0x7f4a5409f0a0 con 0x7f4a54135b50 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a40003520 con 0x7f4a54135b50 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a48001340 con 0x7f4a54135b50 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): dump: 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.634 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.635 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4284830021 0 0) 0x7f4a540a0390 con 0x7f4a54135f70 2024-09-17T15:50:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a5409f0a0 con 0x7f4a54135f70 2024-09-17T15:50:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a4c001a20 con 0x7f4a54135f70 2024-09-17T15:50:02.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:50:02.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient(hunting): dump: 2024-09-17T15:50:02.636 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.636 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.636 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.636 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.636 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.636 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.636 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.636 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.636 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.636 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.636 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2269370208 0 0) 0x7f4a40003520 con 0x7f4a54135b50 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 >> v1:172.21.15.100:6792/0 conn(0x7f4a54135f70 legacy=0x7f4a541d5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 >> v1:172.21.15.64:6791/0 conn(0x7f4a540a5520 legacy=0x7f4a54131860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient: _finish_hunting 0 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 monclient: found mon.b 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a540a8f60 con 0x7f4a54135b50 2024-09-17T15:50:02.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient: _finish_auth 0 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a5a953640 5 monclient: authenticate success, global_id 124256 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a5a953640 10 monclient: _renew_subs 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a5a953640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a5a953640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4a541352e0 con 0x7f4a54135b50 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a5a953640 10 monclient: _renew_subs 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a5a953640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a5a953640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4a540a0670 con 0x7f4a54135b50 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4a480035f0 con 0x7f4a54135b50 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a48003ae0 con 0x7f4a54135b50 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a527fc640 4 set_mon_vals no callback set 2024-09-17T15:50:02.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:02.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.613+0000 7f4a50ff9640 10 monclient: dump: 2024-09-17T15:50:02.639 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.639 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.639 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.639 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.639 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.639 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.639 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.639 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.639 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.639 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.639 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.614+0000 7f4a5a953640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.614+0000 7f4a5a953640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4a340065c0 con 0x7f4a34002860 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.614+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f4a480103c0 con 0x7f4a54135b50 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.614+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(197..197 src has 1..197) ==== 3898+0+0 (unknown 1479433347 0 0) 0x7f4a4804d930 con 0x7f4a54135b50 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.614+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1459872794 0 0) 0x7f4a340065c0 con 0x7f4a34002860 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.614+0000 7f4a58ef0640 10 monclient: get_auth_request con 0x7f4a40040c80 auth_method 0 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.614+0000 7f4a50ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.614+0000 7f4a50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.614+0000 7f4a50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.614+0000 7f4a50ff9640 10 monclient(hunting): my global_id is 5950 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.614+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a40003520 con 0x7f4a34002860 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.615+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1749137777 0 0) 0x7f4a40003520 con 0x7f4a34002860 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.615+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a340065c0 con 0x7f4a34002860 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.615+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a44004fc0 con 0x7f4a34002860 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.615+0000 7f4a50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.615+0000 7f4a50ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.615+0000 7f4a50ff9640 10 monclient: dump: 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.641 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.642 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.615+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3956826740 0 0) 0x7f4a340065c0 con 0x7f4a34002860 2024-09-17T15:50:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.615+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.100:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a3400a160 con 0x7f4a34002860 2024-09-17T15:50:02.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.615+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4a44006540 con 0x7f4a34002860 2024-09-17T15:50:02.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.615+0000 7f4a50ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:02.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.615+0000 7f4a50ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:02.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.615+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 >> v1:172.21.15.100:6789/0 conn(0x7f4a34002860 legacy=0x7f4a34004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.615+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 >> v1:172.21.15.100:6789/0 conn(0x7f4a34002860 legacy=0x7f4a34004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:02.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.621+0000 7f4a5a953640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:02.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.621+0000 7f4a5a953640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4a34009e50 con 0x7f4a34009110 2024-09-17T15:50:02.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3126562687 0 0) 0x7f4a34009e50 con 0x7f4a34009110 2024-09-17T15:50:02.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 10 monclient(hunting): my global_id is 5959 2024-09-17T15:50:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a340065c0 con 0x7f4a34009110 2024-09-17T15:50:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3761803524 0 0) 0x7f4a340065c0 con 0x7f4a34009110 2024-09-17T15:50:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a34009e50 con 0x7f4a34009110 2024-09-17T15:50:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4a44005440 con 0x7f4a34009110 2024-09-17T15:50:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 10 monclient: dump: 2024-09-17T15:50:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.645 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.646 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.646 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.646 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.646 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.646 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.646 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.646 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.646 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.646 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.646 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.646 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2056615899 0 0) 0x7f4a34009e50 con 0x7f4a34009110 2024-09-17T15:50:02.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.622+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 --> v1:172.21.15.100:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4a3400c3f0 con 0x7f4a34009110 2024-09-17T15:50:02.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7546 (unknown 0 0 2792353720) 0x7f4a440057b0 con 0x7f4a34009110 2024-09-17T15:50:02.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a50ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:02.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a50ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:02.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 >> v1:172.21.15.100:6789/0 conn(0x7f4a34009110 legacy=0x7f4a340095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a50ff9640 1 -- 172.21.15.64:0/2024525200 >> v1:172.21.15.100:6789/0 conn(0x7f4a34009110 legacy=0x7f4a340095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:02.647 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "leader", 2024-09-17T15:50:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:50:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 259273, 2024-09-17T15:50:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:50:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:50:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:50:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:50:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:50:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:50:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:50:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:02.667 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a5a953640 1 -- 172.21.15.64:0/2024525200 >> v1:172.21.15.100:6812/4210963484 conn(0x7f4a40040c80 legacy=0x7f4a40043140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a5a953640 10 monclient: shutdown 2024-09-17T15:50:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a5a953640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a5a953640 1 -- 172.21.15.64:0/2024525200 >> v1:172.21.15.64:6790/0 conn(0x7f4a54135b50 legacy=0x7f4a54131f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a58ef0640 1 -- 172.21.15.64:0/2024525200 reap_dead start 2024-09-17T15:50:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a58ef0640 1 -- 172.21.15.64:0/2024525200 reap_dead start 2024-09-17T15:50:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a5a953640 1 -- 172.21.15.64:0/2024525200 shutdown_connections 2024-09-17T15:50:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a5a953640 1 -- 172.21.15.64:0/2024525200 >> 172.21.15.64:0/2024525200 conn(0x7f4a5406db30 msgr2=0x7f4a541334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a5a953640 1 -- 172.21.15.64:0/2024525200 shutdown_connections 2024-09-17T15:50:02.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.623+0000 7f4a5a953640 1 -- 172.21.15.64:0/2024525200 wait complete. 2024-09-17T15:50:02.669 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:50:02.669 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-17T15:50:02.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-17T15:50:02.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.769+0000 7f7e3886b640 10 monclient: get_monmap_and_config 2024-09-17T15:50:02.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.769+0000 7f7e3886b640 10 monclient: build_initial_monmap 2024-09-17T15:50:02.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.769+0000 7f7e3886b640 10 monclient: monmap: 2024-09-17T15:50:02.771 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:02.771 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.771 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:02.770391+0000 2024-09-17T15:50:02.772 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:02.770391+0000 2024-09-17T15:50:02.772 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:02.772 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.772 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:02.772 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:02.772 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:02.772 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:02.772 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:02.772 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:02.772 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:02.772 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:02.772 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:02.772 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.770+0000 7f7e3886b640 1 Processor -- start 2024-09-17T15:50:02.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.770+0000 7f7e3886b640 1 -- start start 2024-09-17T15:50:02.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.770+0000 7f7e3886b640 10 monclient: init 2024-09-17T15:50:02.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.770+0000 7f7e3886b640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:02.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.770+0000 7f7e3886b640 10 monclient: _add_conns ranks=[5,8,3,4,2,1,6,7,0] 2024-09-17T15:50:02.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.770+0000 7f7e3886b640 10 monclient(hunting): picked mon.noname-h con 0x7f7e340a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:50:02.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.770+0000 7f7e3886b640 10 monclient(hunting): picked mon.noname-e con 0x7f7e34135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:50:02.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.770+0000 7f7e3886b640 10 monclient(hunting): picked mon.noname-g con 0x7f7e34135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:02.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.770+0000 7f7e3886b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e3409d6f0 con 0x7f7e34135f70 2024-09-17T15:50:02.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.770+0000 7f7e3886b640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e3409d9d0 con 0x7f7e340a5520 2024-09-17T15:50:02.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e3886b640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e3409dcb0 con 0x7f7e34135b50 2024-09-17T15:50:02.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e3886b640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:02.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e3886b640 10 monclient(hunting): authenticate will time out at 1000.446032s 2024-09-17T15:50:02.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e2bfff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f7e34135b50 0x7f7e3413deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:58162/0 (socket says 172.21.15.64:58162) 2024-09-17T15:50:02.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e2bfff640 1 -- 172.21.15.64:0/3355044241 learned_addr learned my addr 172.21.15.64:0/3355044241 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:02.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 1 -- 172.21.15.64:0/3355044241 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3317419636 0 0) 0x7f7e3409dcb0 con 0x7f7e34135b50 2024-09-17T15:50:02.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient(hunting): my global_id is 5795 2024-09-17T15:50:02.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 1 -- 172.21.15.64:0/3355044241 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e20003520 con 0x7f7e34135b50 2024-09-17T15:50:02.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 1 -- 172.21.15.64:0/3355044241 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 160949639 0 0) 0x7f7e20003520 con 0x7f7e34135b50 2024-09-17T15:50:02.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 1 -- 172.21.15.64:0/3355044241 >> v1:172.21.15.100:6791/0 conn(0x7f7e340a5520 legacy=0x7f7e340a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 1 -- 172.21.15.64:0/3355044241 >> v1:172.21.15.100:6790/0 conn(0x7f7e34135f70 legacy=0x7f7e341415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:50:02.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 1 monclient: found mon.noname-e 2024-09-17T15:50:02.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:02.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:50:02.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 1 -- 172.21.15.64:0/3355044241 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3409ccd0 con 0x7f7e34135b50 2024-09-17T15:50:02.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient: _finish_auth 0 2024-09-17T15:50:02.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:50:02.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:02.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 1 -- 172.21.15.64:0/3355044241 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7e1c001830 con 0x7f7e34135b50 2024-09-17T15:50:02.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e3886b640 5 monclient: authenticate success, global_id 5795 2024-09-17T15:50:02.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e3886b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:02.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:50:02.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient: dump: 2024-09-17T15:50:02.782 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.782 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.784 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e3886b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:02.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 1 -- 172.21.15.64:0/3355044241 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e1c001a50 con 0x7f7e34135b50 2024-09-17T15:50:02.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:02.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 1 -- 172.21.15.64:0/3355044241 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7e1c004be0 con 0x7f7e34135b50 2024-09-17T15:50:02.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e3886b640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:02.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e3886b640 4 set_mon_vals no callback set 2024-09-17T15:50:02.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:50:02.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.771+0000 7f7e31d9b640 10 monclient: dump: 2024-09-17T15:50:02.784 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.784 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.784 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.772+0000 7f7e3886b640 10 monclient: shutdown 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.772+0000 7f7e3886b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.772+0000 7f7e3886b640 1 -- 172.21.15.64:0/3355044241 >> v1:172.21.15.64:6793/0 conn(0x7f7e34135b50 legacy=0x7f7e3413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.772+0000 7f7e3886b640 1 -- 172.21.15.64:0/3355044241 shutdown_connections 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.772+0000 7f7e3886b640 1 -- 172.21.15.64:0/3355044241 >> 172.21.15.64:0/3355044241 conn(0x7f7e3406db30 msgr2=0x7f7e340ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.772+0000 7f7e3886b640 1 -- 172.21.15.64:0/3355044241 shutdown_connections 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.772+0000 7f7e3886b640 1 -- 172.21.15.64:0/3355044241 wait complete. 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.772+0000 7f7e3886b640 10 monclient: build_initial_monmap 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.772+0000 7f7e3886b640 10 monclient: monmap: 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:02.786 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:02.788 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:02.788 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:02.788 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:02.788 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:02.788 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:02.788 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:02.788 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:02.788 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:02.788 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:02.789 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e3886b640 1 Processor -- start 2024-09-17T15:50:02.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e3886b640 1 -- start start 2024-09-17T15:50:02.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e3886b640 10 monclient: init 2024-09-17T15:50:02.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e3886b640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:02.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e3886b640 10 monclient: _add_conns ranks=[7,1,4,3,8,5,6,2,0] 2024-09-17T15:50:02.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e3886b640 10 monclient(hunting): picked mon.noname-i con 0x7f7e340a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:50:02.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e3886b640 10 monclient(hunting): picked mon.noname-f con 0x7f7e34135b50 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:02.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e3886b640 10 monclient(hunting): picked mon.noname-c con 0x7f7e34135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:02.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e3886b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e3413b120 con 0x7f7e34135b50 2024-09-17T15:50:02.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e3886b640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e34134ad0 con 0x7f7e34135f70 2024-09-17T15:50:02.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e3886b640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e340a9320 con 0x7f7e340a5520 2024-09-17T15:50:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e3886b640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e3886b640 10 monclient(hunting): authenticate will time out at 1000.448846s 2024-09-17T15:50:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e32d9d640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f7e34135f70 0x7f7e341d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:52810/0 (socket says 172.21.15.64:52810) 2024-09-17T15:50:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.773+0000 7f7e32d9d640 1 -- 172.21.15.64:0/832659232 learned_addr learned my addr 172.21.15.64:0/832659232 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3025277510 0 0) 0x7f7e34134ad0 con 0x7f7e34135f70 2024-09-17T15:50:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): my global_id is 5863 2024-09-17T15:50:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e14003550 con 0x7f7e34135f70 2024-09-17T15:50:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1548307843 0 0) 0x7f7e340a9320 con 0x7f7e340a5520 2024-09-17T15:50:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): my global_id is 5659 2024-09-17T15:50:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e34134ad0 con 0x7f7e340a5520 2024-09-17T15:50:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3613582701 0 0) 0x7f7e3409ccd0 con 0x7f7e34135b50 2024-09-17T15:50:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): my global_id is 5968 2024-09-17T15:50:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e340a9320 con 0x7f7e34135b50 2024-09-17T15:50:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3235043865 0 0) 0x7f7e14003550 con 0x7f7e34135f70 2024-09-17T15:50:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e3409ccd0 con 0x7f7e34135f70 2024-09-17T15:50:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7e2c001df0 con 0x7f7e34135f70 2024-09-17T15:50:02.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): dump: 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.792 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.793 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.793 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.793 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.793 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:02.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2167359056 0 0) 0x7f7e34134ad0 con 0x7f7e340a5520 2024-09-17T15:50:02.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e14003550 con 0x7f7e340a5520 2024-09-17T15:50:02.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7e24001a40 con 0x7f7e340a5520 2024-09-17T15:50:02.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:50:02.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): dump: 2024-09-17T15:50:02.793 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.793 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.794 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.794 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.794 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.794 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.794 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.794 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.794 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.794 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.794 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.794 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.794 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.794 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.794 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.795 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1790167633 0 0) 0x7f7e340a9320 con 0x7f7e34135b50 2024-09-17T15:50:02.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e34134ad0 con 0x7f7e34135b50 2024-09-17T15:50:02.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7e1c004b20 con 0x7f7e34135b50 2024-09-17T15:50:02.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:02.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient(hunting): dump: 2024-09-17T15:50:02.795 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.795 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.795 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.795 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.795 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.795 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.796 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.796 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.796 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.796 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.796 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.796 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.796 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.796 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.796 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.796 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3115775382 0 0) 0x7f7e3409ccd0 con 0x7f7e34135f70 2024-09-17T15:50:02.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 >> v1:172.21.15.100:6792/0 conn(0x7f7e340a5520 legacy=0x7f7e34136f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 >> v1:172.21.15.100:6789/0 conn(0x7f7e34135b50 legacy=0x7f7e34137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient: _finish_hunting 0 2024-09-17T15:50:02.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 monclient: found mon.c 2024-09-17T15:50:02.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:02.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:50:02.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e341389a0 con 0x7f7e34135f70 2024-09-17T15:50:02.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient: _finish_auth 0 2024-09-17T15:50:02.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient: _check_auth_tickets 2024-09-17T15:50:02.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:02.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e3886b640 5 monclient: authenticate success, global_id 5863 2024-09-17T15:50:02.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e3886b640 10 monclient: _renew_subs 2024-09-17T15:50:02.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e3886b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:50:02.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e3886b640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7e34137d20 con 0x7f7e34135f70 2024-09-17T15:50:02.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e3886b640 10 monclient: _renew_subs 2024-09-17T15:50:02.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e3886b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:50:02.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e3886b640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f7e3413a460 con 0x7f7e34135f70 2024-09-17T15:50:02.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e2c003c40 con 0x7f7e34135f70 2024-09-17T15:50:02.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:02.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7e2c004170 con 0x7f7e34135f70 2024-09-17T15:50:02.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:02.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e3159a640 4 set_mon_vals no callback set 2024-09-17T15:50:02.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.774+0000 7f7e2affd640 10 monclient: dump: 2024-09-17T15:50:02.798 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.798 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.798 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.798 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.799 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.799 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.799 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.799 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.799 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.799 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.799 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.799 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.799 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.799 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.799 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.799 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.775+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f7e2c004390 con 0x7f7e34135f70 2024-09-17T15:50:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.775+0000 7f7e3886b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.775+0000 7f7e3886b640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7e180065c0 con 0x7f7e18002860 2024-09-17T15:50:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.775+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(197..197 src has 1..197) ==== 3898+0+0 (unknown 1479433347 0 0) 0x7f7e2c04efb0 con 0x7f7e34135f70 2024-09-17T15:50:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.775+0000 7f7e2bfff640 10 monclient: get_auth_request con 0x7f7e14040d50 auth_method 0 2024-09-17T15:50:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.775+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1547990904 0 0) 0x7f7e180065c0 con 0x7f7e18002860 2024-09-17T15:50:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.775+0000 7f7e2affd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.775+0000 7f7e2affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.775+0000 7f7e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.775+0000 7f7e2affd640 10 monclient(hunting): my global_id is 114933 2024-09-17T15:50:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.775+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e3409ccd0 con 0x7f7e18002860 2024-09-17T15:50:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.776+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 460859654 0 0) 0x7f7e3409ccd0 con 0x7f7e18002860 2024-09-17T15:50:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.776+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e180065c0 con 0x7f7e18002860 2024-09-17T15:50:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.776+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7e24003b20 con 0x7f7e18002860 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.776+0000 7f7e2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.776+0000 7f7e2affd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.776+0000 7f7e2affd640 10 monclient: dump: 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.802 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.776+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 752154052 0 0) 0x7f7e180065c0 con 0x7f7e18002860 2024-09-17T15:50:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.776+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.100:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e1800a160 con 0x7f7e18002860 2024-09-17T15:50:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.776+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7e24005000 con 0x7f7e18002860 2024-09-17T15:50:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.776+0000 7f7e2affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.776+0000 7f7e2affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.776+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 >> v1:172.21.15.100:6790/0 conn(0x7f7e18002860 legacy=0x7f7e18004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.776+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 >> v1:172.21.15.100:6790/0 conn(0x7f7e18002860 legacy=0x7f7e18004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:02.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.782+0000 7f7e3886b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:02.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.782+0000 7f7e3886b640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7e18009e70 con 0x7f7e18009130 2024-09-17T15:50:02.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3476934320 0 0) 0x7f7e18009e70 con 0x7f7e18009130 2024-09-17T15:50:02.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 10 monclient(hunting): my global_id is 114942 2024-09-17T15:50:02.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e180065c0 con 0x7f7e18009130 2024-09-17T15:50:02.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 784282510 0 0) 0x7f7e180065c0 con 0x7f7e18009130 2024-09-17T15:50:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e18009e70 con 0x7f7e18009130 2024-09-17T15:50:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7e240057a0 con 0x7f7e18009130 2024-09-17T15:50:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:50:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 10 monclient: dump: 2024-09-17T15:50:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.806 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2193886550 0 0) 0x7f7e18009e70 con 0x7f7e18009130 2024-09-17T15:50:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.783+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 --> v1:172.21.15.100:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7e1800c410 con 0x7f7e18009130 2024-09-17T15:50:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1909944365) 0x7f7e24005850 con 0x7f7e18009130 2024-09-17T15:50:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e2affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:02.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e2affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:02.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 >> v1:172.21.15.100:6790/0 conn(0x7f7e18009130 legacy=0x7f7e180095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e2affd640 1 -- 172.21.15.64:0/832659232 >> v1:172.21.15.100:6790/0 conn(0x7f7e18009130 legacy=0x7f7e180095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:02.807 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:02.807 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:02.807 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:02.807 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:50:02.807 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:50:02.807 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 126006, 2024-09-17T15:50:02.807 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:02.808 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:02.808 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:02.808 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:02.808 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:02.808 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:50:02.808 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:50:02.808 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:50:02.808 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:50:02.808 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:50:02.808 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.808 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:50:02.808 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.809 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.810 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:02.810 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:02.810 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.810 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.810 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.810 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.810 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.810 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.810 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.810 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.810 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.810 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.810 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.811 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.811 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:02.811 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:02.811 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:02.811 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:02.811 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:02.811 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:02.811 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:02.811 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:02.811 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:02.811 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:02.811 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:02.812 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:02.812 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:02.812 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:02.812 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:02.812 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:02.812 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:02.812 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.812 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.812 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.812 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.812 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.812 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.812 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.813 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.813 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.813 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.813 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.813 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.813 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:02.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.813 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:02.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.813 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:02.813 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:02.813 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.813 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.815 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:02.815 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:02.815 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.815 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.815 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.815 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:02.815 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.815 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.815 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.815 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.816 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.816 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.816 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.816 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:02.816 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:02.816 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.816 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.816 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.816 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:02.816 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.817 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.817 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.817 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.817 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.817 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.817 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.817 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.817 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:02.817 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:02.817 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.818 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.818 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.818 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:02.818 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.818 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.818 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.818 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.818 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.818 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.818 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.819 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:02.819 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:02.819 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.819 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.819 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.819 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:02.819 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.819 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.820 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:02.821 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.821 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.821 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.821 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.821 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.821 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.821 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.821 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.821 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:02.821 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.823 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.823 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:02.823 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:02.823 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.823 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.823 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.823 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:02.823 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.823 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.824 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.824 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.824 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.824 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.824 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.824 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:02.824 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:02.824 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.824 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.825 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.825 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:02.825 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.825 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.825 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.825 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.825 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.825 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.825 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:02.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.827 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.827 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.827 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:02.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.827 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.827 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:02.827 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:02.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e3886b640 1 -- 172.21.15.64:0/832659232 >> v1:172.21.15.100:6812/4210963484 conn(0x7f7e14040d50 legacy=0x7f7e14043210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e3886b640 10 monclient: shutdown 2024-09-17T15:50:02.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e3886b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e3886b640 1 -- 172.21.15.64:0/832659232 >> v1:172.21.15.64:6791/0 conn(0x7f7e34135f70 legacy=0x7f7e341d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e32d9d640 1 -- 172.21.15.64:0/832659232 reap_dead start 2024-09-17T15:50:02.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e32d9d640 1 -- 172.21.15.64:0/832659232 reap_dead start 2024-09-17T15:50:02.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e3886b640 1 -- 172.21.15.64:0/832659232 shutdown_connections 2024-09-17T15:50:02.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e3886b640 1 -- 172.21.15.64:0/832659232 >> 172.21.15.64:0/832659232 conn(0x7f7e3406db30 msgr2=0x7f7e340a8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.784+0000 7f7e3886b640 1 -- 172.21.15.64:0/832659232 shutdown_connections 2024-09-17T15:50:02.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.785+0000 7f7e3886b640 1 -- 172.21.15.64:0/832659232 wait complete. 2024-09-17T15:50:02.829 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:50:02.830 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-17T15:50:02.830 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-17T15:50:02.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.930+0000 7f3b8bca9640 10 monclient: get_monmap_and_config 2024-09-17T15:50:02.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.930+0000 7f3b8bca9640 10 monclient: build_initial_monmap 2024-09-17T15:50:02.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.930+0000 7f3b8bca9640 10 monclient: monmap: 2024-09-17T15:50:02.932 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:02.932 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.932 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:02.931407+0000 2024-09-17T15:50:02.932 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:02.931407+0000 2024-09-17T15:50:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:02.933 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:02.933 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.931+0000 7f3b8bca9640 1 Processor -- start 2024-09-17T15:50:02.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.931+0000 7f3b8bca9640 1 -- start start 2024-09-17T15:50:02.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.931+0000 7f3b8bca9640 10 monclient: init 2024-09-17T15:50:02.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.931+0000 7f3b8bca9640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:02.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.931+0000 7f3b8bca9640 10 monclient: _add_conns ranks=[7,0,3,1,2,6,8,4,5] 2024-09-17T15:50:02.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.931+0000 7f3b8bca9640 10 monclient(hunting): picked mon.noname-i con 0x7f3b840a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:50:02.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.931+0000 7f3b8bca9640 10 monclient(hunting): picked mon.noname-a con 0x7f3b84135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:50:02.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.931+0000 7f3b8bca9640 10 monclient(hunting): picked mon.noname-g con 0x7f3b84135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:02.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.931+0000 7f3b8bca9640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b8409d6f0 con 0x7f3b84135b50 2024-09-17T15:50:02.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.931+0000 7f3b8bca9640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b8409d9d0 con 0x7f3b84135f70 2024-09-17T15:50:02.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.931+0000 7f3b8bca9640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b8409dcb0 con 0x7f3b840a5520 2024-09-17T15:50:02.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.931+0000 7f3b8bca9640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:02.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.931+0000 7f3b8bca9640 10 monclient(hunting): authenticate will time out at 1000.606912s 2024-09-17T15:50:02.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b89244640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f3b84135b50 0x7f3b8413deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35236/0 (socket says 172.21.15.64:35236) 2024-09-17T15:50:02.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b89244640 1 -- 172.21.15.64:0/1755058635 learned_addr learned my addr 172.21.15.64:0/1755058635 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:02.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 -- 172.21.15.64:0/1755058635 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2181443586 0 0) 0x7f3b8409d6f0 con 0x7f3b84135b50 2024-09-17T15:50:02.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient(hunting): my global_id is 124200 2024-09-17T15:50:02.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 -- 172.21.15.64:0/1755058635 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b70003520 con 0x7f3b84135b50 2024-09-17T15:50:02.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 -- 172.21.15.64:0/1755058635 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2685020401 0 0) 0x7f3b8409dcb0 con 0x7f3b840a5520 2024-09-17T15:50:02.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient(hunting): my global_id is 5668 2024-09-17T15:50:02.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 -- 172.21.15.64:0/1755058635 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b8409d6f0 con 0x7f3b840a5520 2024-09-17T15:50:02.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 -- 172.21.15.64:0/1755058635 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3823442557 0 0) 0x7f3b8409d9d0 con 0x7f3b84135f70 2024-09-17T15:50:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient(hunting): my global_id is 114951 2024-09-17T15:50:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 -- 172.21.15.64:0/1755058635 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b8409dcb0 con 0x7f3b84135f70 2024-09-17T15:50:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 -- 172.21.15.64:0/1755058635 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1454788960 0 0) 0x7f3b70003520 con 0x7f3b84135b50 2024-09-17T15:50:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 -- 172.21.15.64:0/1755058635 >> v1:172.21.15.100:6792/0 conn(0x7f3b840a5520 legacy=0x7f3b840a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 -- 172.21.15.64:0/1755058635 >> v1:172.21.15.100:6790/0 conn(0x7f3b84135f70 legacy=0x7f3b841415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient: _finish_hunting 0 2024-09-17T15:50:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 monclient: found mon.noname-a 2024-09-17T15:50:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:50:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 -- 172.21.15.64:0/1755058635 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8409ccd0 con 0x7f3b84135b50 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient: _finish_auth 0 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient: _check_auth_tickets 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 -- 172.21.15.64:0/1755058635 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3b74001830 con 0x7f3b84135b50 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b8bca9640 5 monclient: authenticate success, global_id 124200 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b8bca9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient: dump: 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.946 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.947 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.948 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b8bca9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 -- 172.21.15.64:0/1755058635 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3b74001a50 con 0x7f3b84135b50 2024-09-17T15:50:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 1 -- 172.21.15.64:0/1755058635 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3b74004be0 con 0x7f3b84135b50 2024-09-17T15:50:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b88a43640 10 monclient: dump: 2024-09-17T15:50:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.948 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.950 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b8bca9640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.932+0000 7f3b8bca9640 4 set_mon_vals no callback set 2024-09-17T15:50:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.933+0000 7f3b8bca9640 10 monclient: shutdown 2024-09-17T15:50:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.933+0000 7f3b8bca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.933+0000 7f3b8bca9640 1 -- 172.21.15.64:0/1755058635 >> v1:172.21.15.64:6789/0 conn(0x7f3b84135b50 legacy=0x7f3b8413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.933+0000 7f3b8bca9640 1 -- 172.21.15.64:0/1755058635 shutdown_connections 2024-09-17T15:50:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.933+0000 7f3b8bca9640 1 -- 172.21.15.64:0/1755058635 >> 172.21.15.64:0/1755058635 conn(0x7f3b8406db30 msgr2=0x7f3b840ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.933+0000 7f3b8bca9640 1 -- 172.21.15.64:0/1755058635 shutdown_connections 2024-09-17T15:50:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.933+0000 7f3b8bca9640 1 -- 172.21.15.64:0/1755058635 wait complete. 2024-09-17T15:50:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.933+0000 7f3b8bca9640 10 monclient: build_initial_monmap 2024-09-17T15:50:02.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.933+0000 7f3b8bca9640 10 monclient: monmap: 2024-09-17T15:50:02.951 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:02.951 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:02.951 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:02.951 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:02.951 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:02.951 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:02.951 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:02.951 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:02.951 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:02.951 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:02.951 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:02.951 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:02.952 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:02.952 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:02.952 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:02.952 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8bca9640 1 Processor -- start 2024-09-17T15:50:02.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8bca9640 1 -- start start 2024-09-17T15:50:02.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8bca9640 10 monclient: init 2024-09-17T15:50:02.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8bca9640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:02.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8bca9640 10 monclient: _add_conns ranks=[1,4,0,6,7,5,3,2,8] 2024-09-17T15:50:02.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8bca9640 10 monclient(hunting): picked mon.noname-f con 0x7f3b840a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:02.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8bca9640 10 monclient(hunting): picked mon.noname-c con 0x7f3b84135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8bca9640 10 monclient(hunting): picked mon.noname-a con 0x7f3b84135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:50:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8bca9640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b84138900 con 0x7f3b84135f70 2024-09-17T15:50:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8bca9640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b840a8fe0 con 0x7f3b840a5520 2024-09-17T15:50:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8bca9640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b8413a3c0 con 0x7f3b84135b50 2024-09-17T15:50:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8bca9640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8bca9640 10 monclient(hunting): authenticate will time out at 1000.609711s 2024-09-17T15:50:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8a246640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f3b84135f70 0x7f3b84137c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35246/0 (socket says 172.21.15.64:35246) 2024-09-17T15:50:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b8a246640 1 -- 172.21.15.64:0/3896173137 learned_addr learned my addr 172.21.15.64:0/3896173137 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.934+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1307300693 0 0) 0x7f3b84138900 con 0x7f3b84135f70 2024-09-17T15:50:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): my global_id is 124209 2024-09-17T15:50:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b68003520 con 0x7f3b84135f70 2024-09-17T15:50:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1207642126 0 0) 0x7f3b8413a3c0 con 0x7f3b84135b50 2024-09-17T15:50:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): my global_id is 5872 2024-09-17T15:50:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b84138900 con 0x7f3b84135b50 2024-09-17T15:50:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1649997559 0 0) 0x7f3b840a8fe0 con 0x7f3b840a5520 2024-09-17T15:50:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): my global_id is 5977 2024-09-17T15:50:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b8413a3c0 con 0x7f3b840a5520 2024-09-17T15:50:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2701612762 0 0) 0x7f3b68003520 con 0x7f3b84135f70 2024-09-17T15:50:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b840a8fe0 con 0x7f3b84135f70 2024-09-17T15:50:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3b7c001e10 con 0x7f3b84135f70 2024-09-17T15:50:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): dump: 2024-09-17T15:50:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.955 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.956 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2954086622 0 0) 0x7f3b8413a3c0 con 0x7f3b840a5520 2024-09-17T15:50:02.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b68003520 con 0x7f3b840a5520 2024-09-17T15:50:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 558028818 0 0) 0x7f3b84138900 con 0x7f3b84135b50 2024-09-17T15:50:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b8413a3c0 con 0x7f3b84135b50 2024-09-17T15:50:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3b80001a10 con 0x7f3b840a5520 2024-09-17T15:50:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): dump: 2024-09-17T15:50:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.957 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.958 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3b74004b20 con 0x7f3b84135b50 2024-09-17T15:50:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient(hunting): dump: 2024-09-17T15:50:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.959 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.960 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.960 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.960 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.960 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.960 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1672663797 0 0) 0x7f3b840a8fe0 con 0x7f3b84135f70 2024-09-17T15:50:02.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 >> v1:172.21.15.64:6791/0 conn(0x7f3b84135b50 legacy=0x7f3b84137570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 >> v1:172.21.15.100:6789/0 conn(0x7f3b840a5520 legacy=0x7f3b84136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:50:02.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 monclient: found mon.a 2024-09-17T15:50:02.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:02.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:50:02.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8413a6a0 con 0x7f3b84135f70 2024-09-17T15:50:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:50:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:50:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b8bca9640 5 monclient: authenticate success, global_id 124209 2024-09-17T15:50:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b8bca9640 10 monclient: _renew_subs 2024-09-17T15:50:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b8bca9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:50:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b8bca9640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3b8413b080 con 0x7f3b84135f70 2024-09-17T15:50:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b8bca9640 10 monclient: _renew_subs 2024-09-17T15:50:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b8bca9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:50:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b8bca9640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3b841d7160 con 0x7f3b84135f70 2024-09-17T15:50:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3b7c003c40 con 0x7f3b84135f70 2024-09-17T15:50:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:02.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3b7c0041b0 con 0x7f3b84135f70 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7bfff640 4 set_mon_vals no callback set 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.935+0000 7f3b7a7fc640 10 monclient: dump: 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.962 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.936+0000 7f3b8bca9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.936+0000 7f3b8bca9640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3b6c0065c0 con 0x7f3b6c002860 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.936+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f3b7c0043d0 con 0x7f3b84135f70 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.936+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(197..197 src has 1..197) ==== 3898+0+0 (unknown 1479433347 0 0) 0x7f3b7c04eff0 con 0x7f3b84135f70 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.936+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 453442421 0 0) 0x7f3b6c0065c0 con 0x7f3b6c002860 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.936+0000 7f3b7a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.936+0000 7f3b7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.936+0000 7f3b7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.936+0000 7f3b7a7fc640 10 monclient(hunting): my global_id is 124286 2024-09-17T15:50:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.936+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b840a8fe0 con 0x7f3b6c002860 2024-09-17T15:50:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.936+0000 7f3b89244640 10 monclient: get_auth_request con 0x7f3b68040d50 auth_method 0 2024-09-17T15:50:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.937+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2290807666 0 0) 0x7f3b840a8fe0 con 0x7f3b6c002860 2024-09-17T15:50:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.937+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b6c0065c0 con 0x7f3b6c002860 2024-09-17T15:50:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.937+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3b80003930 con 0x7f3b6c002860 2024-09-17T15:50:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.937+0000 7f3b7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.937+0000 7f3b7a7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:50:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.937+0000 7f3b7a7fc640 10 monclient: dump: 2024-09-17T15:50:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.964 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.937+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 319839602 0 0) 0x7f3b6c0065c0 con 0x7f3b6c002860 2024-09-17T15:50:02.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.937+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.100:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b6c00a160 con 0x7f3b6c002860 2024-09-17T15:50:02.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.937+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3b80005b10 con 0x7f3b6c002860 2024-09-17T15:50:02.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.937+0000 7f3b7a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:02.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.937+0000 7f3b7a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:02.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.937+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 >> v1:172.21.15.100:6791/0 conn(0x7f3b6c002860 legacy=0x7f3b6c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.937+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 >> v1:172.21.15.100:6791/0 conn(0x7f3b6c002860 legacy=0x7f3b6c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:02.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.943+0000 7f3b8bca9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:02.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.943+0000 7f3b8bca9640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3b6c009e00 con 0x7f3b6c0090e0 2024-09-17T15:50:02.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 220714010 0 0) 0x7f3b6c009e00 con 0x7f3b6c0090e0 2024-09-17T15:50:02.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:02.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:02.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:02.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 10 monclient(hunting): my global_id is 124295 2024-09-17T15:50:02.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b6c0065c0 con 0x7f3b6c0090e0 2024-09-17T15:50:02.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 611089011 0 0) 0x7f3b6c0065c0 con 0x7f3b6c0090e0 2024-09-17T15:50:02.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b6c009e00 con 0x7f3b6c0090e0 2024-09-17T15:50:02.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3b800041c0 con 0x7f3b6c0090e0 2024-09-17T15:50:02.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:02.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:50:02.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 10 monclient: dump: 2024-09-17T15:50:02.968 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:02.968 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:02.968 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:02.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 802194676 0 0) 0x7f3b6c009e00 con 0x7f3b6c0090e0 2024-09-17T15:50:02.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 --> v1:172.21.15.100:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3b6c00c3a0 con 0x7f3b6c0090e0 2024-09-17T15:50:02.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 157051968) 0x7f3b800056f0 con 0x7f3b6c0090e0 2024-09-17T15:50:02.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:02.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:02.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 >> v1:172.21.15.100:6791/0 conn(0x7f3b6c0090e0 legacy=0x7f3b6c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.944+0000 7f3b7a7fc640 1 -- 172.21.15.64:0/3896173137 >> v1:172.21.15.100:6791/0 conn(0x7f3b6c0090e0 legacy=0x7f3b6c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:02.970 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 9867, 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.971 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:50:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.972 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.973 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:02.974 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:02.975 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:02.976 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.977 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.978 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.979 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:02.980 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.981 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.984 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.985 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.986 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:02.987 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:02.988 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:02.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:02.990 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:02.993 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:02.993 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:02.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.945+0000 7f3b8bca9640 1 -- 172.21.15.64:0/3896173137 >> v1:172.21.15.100:6812/4210963484 conn(0x7f3b68040d50 legacy=0x7f3b68043210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.945+0000 7f3b8bca9640 10 monclient: shutdown 2024-09-17T15:50:02.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.945+0000 7f3b8bca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.945+0000 7f3b8bca9640 1 -- 172.21.15.64:0/3896173137 >> v1:172.21.15.64:6789/0 conn(0x7f3b84135f70 legacy=0x7f3b84137c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.945+0000 7f3b8a246640 1 -- 172.21.15.64:0/3896173137 reap_dead start 2024-09-17T15:50:02.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.945+0000 7f3b8a246640 1 -- 172.21.15.64:0/3896173137 reap_dead start 2024-09-17T15:50:02.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.945+0000 7f3b8bca9640 1 -- 172.21.15.64:0/3896173137 shutdown_connections 2024-09-17T15:50:02.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.945+0000 7f3b8bca9640 1 -- 172.21.15.64:0/3896173137 >> 172.21.15.64:0/3896173137 conn(0x7f3b8406db30 msgr2=0x7f3b84133fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.945+0000 7f3b8bca9640 1 -- 172.21.15.64:0/3896173137 shutdown_connections 2024-09-17T15:50:02.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:02.945+0000 7f3b8bca9640 1 -- 172.21.15.64:0/3896173137 wait complete. 2024-09-17T15:50:02.995 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:50:02.996 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-17T15:50:02.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-17T15:50:03.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.096+0000 7efd6bee2640 10 monclient: get_monmap_and_config 2024-09-17T15:50:03.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.096+0000 7efd6bee2640 10 monclient: build_initial_monmap 2024-09-17T15:50:03.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.096+0000 7efd6bee2640 10 monclient: monmap: 2024-09-17T15:50:03.098 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:03.098 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.099 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:03.097520+0000 2024-09-17T15:50:03.099 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:03.097520+0000 2024-09-17T15:50:03.099 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:03.099 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.099 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:03.099 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:03.099 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:03.099 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:03.099 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:03.099 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:03.100 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:03.100 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:03.100 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:03.100 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.097+0000 7efd6bee2640 1 Processor -- start 2024-09-17T15:50:03.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.097+0000 7efd6bee2640 1 -- start start 2024-09-17T15:50:03.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.097+0000 7efd6bee2640 10 monclient: init 2024-09-17T15:50:03.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6bee2640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:03.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6bee2640 10 monclient: _add_conns ranks=[4,2,1,8,7,5,0,3,6] 2024-09-17T15:50:03.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6bee2640 10 monclient(hunting): picked mon.noname-c con 0x7efd6409b3b0 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:03.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6bee2640 10 monclient(hunting): picked mon.noname-b con 0x7efd640a40d0 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:03.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6bee2640 10 monclient(hunting): picked mon.noname-f con 0x7efd6413bc00 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:03.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6bee2640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd640a0db0 con 0x7efd6413bc00 2024-09-17T15:50:03.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6bee2640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd640a1090 con 0x7efd640a40d0 2024-09-17T15:50:03.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6bee2640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd640a1370 con 0x7efd6409b3b0 2024-09-17T15:50:03.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6bee2640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:03.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6bee2640 10 monclient(hunting): authenticate will time out at 1000.773240s 2024-09-17T15:50:03.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd69c7e640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7efd6409b3b0 0x7efd640a2030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:52820/0 (socket says 172.21.15.64:52820) 2024-09-17T15:50:03.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd69c7e640 1 -- 172.21.15.64:0/3853954985 learned_addr learned my addr 172.21.15.64:0/3853954985 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:03.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6947d640 1 --1- 172.21.15.64:0/3853954985 >> v1:172.21.15.64:6790/0 conn(0x7efd640a40d0 0x7efd6409b790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:47038/0 (socket says 172.21.15.64:47038) 2024-09-17T15:50:03.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 1 -- 172.21.15.64:0/3853954985 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 818212424 0 0) 0x7efd640a1370 con 0x7efd6409b3b0 2024-09-17T15:50:03.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient(hunting): _negotiate 2024-09-17T15:50:03.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:03.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:03.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient(hunting): my global_id is 5881 2024-09-17T15:50:03.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 1 -- 172.21.15.64:0/3853954985 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd50003520 con 0x7efd6409b3b0 2024-09-17T15:50:03.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 1 -- 172.21.15.64:0/3853954985 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2547187505 0 0) 0x7efd640a1090 con 0x7efd640a40d0 2024-09-17T15:50:03.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient(hunting): _negotiate 2024-09-17T15:50:03.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:03.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:03.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient(hunting): my global_id is 124265 2024-09-17T15:50:03.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 1 -- 172.21.15.64:0/3853954985 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd640a1370 con 0x7efd640a40d0 2024-09-17T15:50:03.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 1 -- 172.21.15.64:0/3853954985 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1490195917 0 0) 0x7efd640a0db0 con 0x7efd6413bc00 2024-09-17T15:50:03.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient(hunting): _negotiate 2024-09-17T15:50:03.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:03.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:03.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient(hunting): my global_id is 5986 2024-09-17T15:50:03.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 1 -- 172.21.15.64:0/3853954985 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd640a1090 con 0x7efd6413bc00 2024-09-17T15:50:03.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 1 -- 172.21.15.64:0/3853954985 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3831559984 0 0) 0x7efd50003520 con 0x7efd6409b3b0 2024-09-17T15:50:03.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 1 -- 172.21.15.64:0/3853954985 >> v1:172.21.15.64:6790/0 conn(0x7efd640a40d0 legacy=0x7efd6409b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:03.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 1 -- 172.21.15.64:0/3853954985 >> v1:172.21.15.100:6789/0 conn(0x7efd6413bc00 legacy=0x7efd6413dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:03.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient: _finish_hunting 0 2024-09-17T15:50:03.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 1 monclient: found mon.noname-c 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 1 -- 172.21.15.64:0/3853954985 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6409bea0 con 0x7efd6409b3b0 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient: _finish_auth 0 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient: _check_auth_tickets 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 1 -- 172.21.15.64:0/3853954985 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7efd60001830 con 0x7efd6409b3b0 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6bee2640 5 monclient: authenticate success, global_id 5881 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6bee2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd68c7c640 10 monclient: dump: 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.113 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.098+0000 7efd6bee2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd68c7c640 1 -- 172.21.15.64:0/3853954985 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd600019d0 con 0x7efd6409b3b0 2024-09-17T15:50:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd68c7c640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd68c7c640 1 -- 172.21.15.64:0/3853954985 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7efd60004be0 con 0x7efd6409b3b0 2024-09-17T15:50:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd68c7c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd68c7c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd68c7c640 10 monclient: dump: 2024-09-17T15:50:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd6bee2640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd6bee2640 4 set_mon_vals no callback set 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd6bee2640 10 monclient: shutdown 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd6bee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd6bee2640 1 -- 172.21.15.64:0/3853954985 >> v1:172.21.15.64:6791/0 conn(0x7efd6409b3b0 legacy=0x7efd640a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd6bee2640 1 -- 172.21.15.64:0/3853954985 shutdown_connections 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd6bee2640 1 -- 172.21.15.64:0/3853954985 >> 172.21.15.64:0/3853954985 conn(0x7efd640a4ef0 msgr2=0x7efd640a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd6bee2640 1 -- 172.21.15.64:0/3853954985 shutdown_connections 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.099+0000 7efd6bee2640 1 -- 172.21.15.64:0/3853954985 wait complete. 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.100+0000 7efd6bee2640 10 monclient: build_initial_monmap 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.100+0000 7efd6bee2640 10 monclient: monmap: 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:03.117 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.100+0000 7efd6bee2640 1 Processor -- start 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.100+0000 7efd6bee2640 1 -- start start 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.100+0000 7efd6bee2640 10 monclient: init 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd6bee2640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd6bee2640 10 monclient: _add_conns ranks=[6,4,2,7,1,0,8,5,3] 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd6bee2640 10 monclient(hunting): picked mon.noname-d con 0x7efd6409b3b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:50:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd6bee2640 10 monclient(hunting): picked mon.noname-c con 0x7efd640a40d0 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd6bee2640 10 monclient(hunting): picked mon.noname-b con 0x7efd6413bc00 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd6bee2640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd64074fa0 con 0x7efd6413bc00 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd6bee2640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd640737a0 con 0x7efd640a40d0 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd6bee2640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd64072c80 con 0x7efd6409b3b0 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd6bee2640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd6bee2640 10 monclient(hunting): authenticate will time out at 1000.776233s 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd69c7e640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7efd6409b3b0 0x7efd641c9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:44084/0 (socket says 172.21.15.64:44084) 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd6947d640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7efd640a40d0 0x7efd64134820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:52822/0 (socket says 172.21.15.64:52822) 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd69c7e640 1 -- 172.21.15.64:0/3173550276 learned_addr learned my addr 172.21.15.64:0/3173550276 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2116291380 0 0) 0x7efd64072c80 con 0x7efd6409b3b0 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): my global_id is 124233 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd48003520 con 0x7efd6409b3b0 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1996064199 0 0) 0x7efd64074fa0 con 0x7efd6413bc00 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): my global_id is 124274 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd64072c80 con 0x7efd6413bc00 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3005703988 0 0) 0x7efd640737a0 con 0x7efd640a40d0 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): my global_id is 5890 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd64074fa0 con 0x7efd640a40d0 2024-09-17T15:50:03.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3666432844 0 0) 0x7efd64072c80 con 0x7efd6413bc00 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd640737a0 con 0x7efd6413bc00 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7efd5c001cf0 con 0x7efd6413bc00 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): dump: 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:03.121 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2114661594 0 0) 0x7efd48003520 con 0x7efd6409b3b0 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd64072c80 con 0x7efd6409b3b0 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1705664017 0 0) 0x7efd64074fa0 con 0x7efd640a40d0 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd48003520 con 0x7efd640a40d0 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7efd60004be0 con 0x7efd6409b3b0 2024-09-17T15:50:03.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): dump: 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:03.123 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7efd54001480 con 0x7efd640a40d0 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.101+0000 7efd5a7fc640 10 monclient(hunting): dump: 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:03.124 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.125 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:03.125 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:03.125 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:03.125 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:03.125 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:03.125 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:03.125 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:03.125 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:03.125 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:03.125 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2712897412 0 0) 0x7efd640737a0 con 0x7efd6413bc00 2024-09-17T15:50:03.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 >> v1:172.21.15.64:6792/0 conn(0x7efd6409b3b0 legacy=0x7efd641c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:03.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 >> v1:172.21.15.64:6791/0 conn(0x7efd640a40d0 legacy=0x7efd64134820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:03.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:50:03.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 1 monclient: found mon.b 2024-09-17T15:50:03.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:03.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:03.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd64134f30 con 0x7efd6413bc00 2024-09-17T15:50:03.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:50:03.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:50:03.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:03.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd6bee2640 5 monclient: authenticate success, global_id 124274 2024-09-17T15:50:03.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd6bee2640 10 monclient: _renew_subs 2024-09-17T15:50:03.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd6bee2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:03.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd6bee2640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efd641351c0 con 0x7efd6413bc00 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd6bee2640 10 monclient: _renew_subs 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd6bee2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd6bee2640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7efd641d3b70 con 0x7efd6413bc00 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd5c003990 con 0x7efd6413bc00 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7efd5c003e80 con 0x7efd6413bc00 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5bfff640 4 set_mon_vals no callback set 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd5a7fc640 10 monclient: dump: 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.127 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.128 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:03.128 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.128 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:03.128 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:03.128 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:03.128 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:03.128 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:03.128 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:03.128 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:03.128 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:03.128 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:03.128 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd6bee2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.102+0000 7efd6bee2640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efd4c0065c0 con 0x7efd4c002860 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7efd5c010760 con 0x7efd6413bc00 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(197..197 src has 1..197) ==== 3898+0+0 (unknown 1479433347 0 0) 0x7efd5c04db20 con 0x7efd6413bc00 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2892275740 0 0) 0x7efd4c0065c0 con 0x7efd4c002860 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 10 monclient(hunting): my global_id is 5677 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd640737a0 con 0x7efd4c002860 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd6947d640 10 monclient: get_auth_request con 0x7efd48040d20 auth_method 0 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3630859662 0 0) 0x7efd640737a0 con 0x7efd4c002860 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd4c0065c0 con 0x7efd4c002860 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7efd60001900 con 0x7efd4c002860 2024-09-17T15:50:03.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 10 monclient: dump: 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:03.131 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:03.131 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:03.131 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:03.131 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3923545758 0 0) 0x7efd4c0065c0 con 0x7efd4c002860 2024-09-17T15:50:03.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.103+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.100:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd4c00a160 con 0x7efd4c002860 2024-09-17T15:50:03.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.104+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7efd60004fb0 con 0x7efd4c002860 2024-09-17T15:50:03.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.104+0000 7efd5a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:03.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.104+0000 7efd5a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:03.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.104+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 >> v1:172.21.15.100:6792/0 conn(0x7efd4c002860 legacy=0x7efd4c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:03.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.104+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 >> v1:172.21.15.100:6792/0 conn(0x7efd4c002860 legacy=0x7efd4c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:03.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.110+0000 7efd6bee2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:03.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.110+0000 7efd6bee2640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efd4c009e50 con 0x7efd4c009110 2024-09-17T15:50:03.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 339341476 0 0) 0x7efd4c009e50 con 0x7efd4c009110 2024-09-17T15:50:03.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:03.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:03.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:03.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 10 monclient(hunting): my global_id is 5686 2024-09-17T15:50:03.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd4c0065c0 con 0x7efd4c009110 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1981053867 0 0) 0x7efd4c0065c0 con 0x7efd4c009110 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd4c009e50 con 0x7efd4c009110 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7efd600057a0 con 0x7efd4c009110 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 10 monclient: dump: 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:03.134 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2976573593 0 0) 0x7efd4c009e50 con 0x7efd4c009110 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 --> v1:172.21.15.100:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efd4c00c3f0 con 0x7efd4c009110 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1934044247) 0x7efd60006bd0 con 0x7efd4c009110 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:03.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 >> v1:172.21.15.100:6792/0 conn(0x7efd4c009110 legacy=0x7efd4c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:03.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.111+0000 7efd5a7fc640 1 -- 172.21.15.64:0/3173550276 >> v1:172.21.15.100:6792/0 conn(0x7efd4c009110 legacy=0x7efd4c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:03.136 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:03.136 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:03.136 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:03.136 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:50:03.136 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 24, 2024-09-17T15:50:03.136 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 259746, 2024-09-17T15:50:03.136 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:03.136 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:03.137 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:03.137 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:03.137 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:03.137 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:50:03.137 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:50:03.137 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:50:03.137 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:50:03.137 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:50:03.137 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:03.137 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:50:03.137 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:03.137 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:03.137 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:03.138 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:03.138 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:03.138 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:03.138 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:03.138 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:03.138 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:03.138 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:03.138 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:03.138 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:03.138 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:03.138 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:03.138 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:03.138 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:03.139 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:03.139 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:03.139 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:03.139 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:03.139 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:03.139 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:03.139 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:03.139 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:03.139 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:03.139 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:03.139 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:03.139 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:03.140 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:03.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.140 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:03.140 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:03.140 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:03.140 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:03.140 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:03.140 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:03.140 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:03.140 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:03.140 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:03.140 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:03.141 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:03.141 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:03.141 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:03.141 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:03.141 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:03.141 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:03.141 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:03.141 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:03.141 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:03.141 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:03.141 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:03.141 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:03.141 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:03.142 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:03.142 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:03.142 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:03.142 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:03.142 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:03.142 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:03.142 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:03.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.142 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:03.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.142 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:03.142 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:03.143 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:03.143 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:03.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.143 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:03.143 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:03.143 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:03.143 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:03.143 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:03.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.143 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:03.143 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:03.143 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:03.143 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:03.144 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:03.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.144 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:03.144 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:03.144 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:03.144 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:03.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.144 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:03.144 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:03.146 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.147 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:03.148 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:03.149 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:03.149 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:03.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.149 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:03.149 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:03.149 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:03.149 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:03.149 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:03.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.149 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:03.149 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:03.150 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:03.150 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:03.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.150 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:03.150 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:03.150 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:03.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:03.150 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:03.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.150 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:03.150 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:03.150 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:03.150 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:03.151 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:03.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.151 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:03.151 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:03.151 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:03.151 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:03.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.151 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:03.151 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:03.151 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:03.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:03.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.153 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:03.154 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:03.154 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:03.154 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:03.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.154 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:03.154 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:03.154 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:03.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:03.154 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:03.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.154 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:03.154 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:03.154 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:03.155 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:03.155 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:03.155 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:03.155 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:03.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.155 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:03.155 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:03.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.155 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:03.155 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:03.155 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:03.155 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:03.155 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:03.156 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:03.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.156 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:03.156 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:03.156 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 4 2024-09-17T15:50:03.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:03.156 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:03.156 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": [ 2024-09-17T15:50:03.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:03.156 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:03.156 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:03.156 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:03.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:03.157 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:03.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:03.157 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:03.157 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:03.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.112+0000 7efd6bee2640 1 -- 172.21.15.64:0/3173550276 >> v1:172.21.15.100:6812/4210963484 conn(0x7efd48040d20 legacy=0x7efd480431e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:03.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.112+0000 7efd6bee2640 10 monclient: shutdown 2024-09-17T15:50:03.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.112+0000 7efd6bee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:03.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.112+0000 7efd6bee2640 1 -- 172.21.15.64:0/3173550276 >> v1:172.21.15.64:6790/0 conn(0x7efd6413bc00 legacy=0x7efd641d2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:03.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.112+0000 7efd6a47f640 1 -- 172.21.15.64:0/3173550276 reap_dead start 2024-09-17T15:50:03.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.112+0000 7efd6a47f640 1 -- 172.21.15.64:0/3173550276 reap_dead start 2024-09-17T15:50:03.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.112+0000 7efd6bee2640 1 -- 172.21.15.64:0/3173550276 shutdown_connections 2024-09-17T15:50:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.112+0000 7efd6bee2640 1 -- 172.21.15.64:0/3173550276 >> 172.21.15.64:0/3173550276 conn(0x7efd640a4ef0 msgr2=0x7efd640a0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.112+0000 7efd6bee2640 1 -- 172.21.15.64:0/3173550276 shutdown_connections 2024-09-17T15:50:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.112+0000 7efd6bee2640 1 -- 172.21.15.64:0/3173550276 wait complete. 2024-09-17T15:50:03.159 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:50:03.159 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['i', 'e', 'h', 'c'] 2024-09-17T15:50:03.159 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:50:03.159 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i 2024-09-17T15:50:03.159 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-09-17T15:50:03.160 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-09-17T15:50:03.160 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:50:03.175 INFO:tasks.ceph.mon.i:Stopped 2024-09-17T15:50:03.175 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-09-17T15:50:03.175 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-09-17T15:50:03.175 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-09-17T15:50:03.175 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:50:03.187 INFO:tasks.ceph.mon.e:Stopped 2024-09-17T15:50:03.187 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h 2024-09-17T15:50:03.187 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-09-17T15:50:03.187 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-09-17T15:50:03.187 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:50:03.227 INFO:tasks.ceph.mon.h:Stopped 2024-09-17T15:50:03.227 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-17T15:50:03.227 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-17T15:50:03.227 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T15:50:03.227 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:50:03.240 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T15:50:03.240 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-09-17T15:50:03.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:50:03.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.378+0000 7fda7eacd640 10 monclient: get_monmap_and_config 2024-09-17T15:50:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.378+0000 7fda7eacd640 10 monclient: build_initial_monmap 2024-09-17T15:50:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.379+0000 7fda7eacd640 10 monclient: monmap: 2024-09-17T15:50:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:03.379859+0000 2024-09-17T15:50:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:03.379859+0000 2024-09-17T15:50:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:03.381 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:03.382 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:03.382 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.379+0000 7fda7eacd640 1 Processor -- start 2024-09-17T15:50:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.379+0000 7fda7eacd640 1 -- start start 2024-09-17T15:50:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.379+0000 7fda7eacd640 10 monclient: init 2024-09-17T15:50:03.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda7eacd640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda7eacd640 10 monclient: _add_conns ranks=[5,4,1,3,2,8,7,0,6] 2024-09-17T15:50:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda7eacd640 10 monclient(hunting): picked mon.noname-h con 0x7fda780a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:50:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda7eacd640 10 monclient(hunting): picked mon.noname-c con 0x7fda78135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda7eacd640 10 monclient(hunting): picked mon.noname-f con 0x7fda78135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda7eacd640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda7809d6f0 con 0x7fda78135f70 2024-09-17T15:50:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda7eacd640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda7809d9d0 con 0x7fda78135b50 2024-09-17T15:50:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda7eacd640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda7809dcb0 con 0x7fda780a5520 2024-09-17T15:50:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda7eacd640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda7eacd640 10 monclient(hunting): authenticate will time out at 1001.055349s 2024-09-17T15:50:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda77fff640 1 -- >> v1:172.21.15.64:6791/0 conn(0x7fda78135b50 legacy=0x7fda7813deb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda7c869640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7fda780a5520 legacy=0x7fda780a5900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:50:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda7d06a640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fda78135f70 0x7fda781415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.64:35768/0 (socket says 172.21.15.64:35768) 2024-09-17T15:50:03.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda7d06a640 1 -- 172.21.15.64:0/100775312 learned_addr learned my addr 172.21.15.64:0/100775312 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda777fe640 1 -- 172.21.15.64:0/100775312 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4027237115 0 0) 0x7fda7809d6f0 con 0x7fda78135f70 2024-09-17T15:50:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda777fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda777fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda777fe640 10 monclient(hunting): my global_id is 5995 2024-09-17T15:50:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.380+0000 7fda777fe640 1 -- 172.21.15.64:0/100775312 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda68003520 con 0x7fda78135f70 2024-09-17T15:50:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 1 -- 172.21.15.64:0/100775312 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3773887681 0 0) 0x7fda68003520 con 0x7fda78135f70 2024-09-17T15:50:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 1 -- 172.21.15.64:0/100775312 >> v1:172.21.15.100:6791/0 conn(0x7fda780a5520 legacy=0x7fda780a5900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 1 -- 172.21.15.64:0/100775312 >> v1:172.21.15.64:6791/0 conn(0x7fda78135b50 legacy=0x7fda7813deb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 10 monclient: _finish_hunting 0 2024-09-17T15:50:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 1 monclient: found mon.noname-f 2024-09-17T15:50:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:03.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.100:6789/0 2024-09-17T15:50:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 1 -- 172.21.15.64:0/100775312 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda7809ccd0 con 0x7fda78135f70 2024-09-17T15:50:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 10 monclient: _finish_auth 0 2024-09-17T15:50:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 10 monclient: _check_auth_tickets 2024-09-17T15:50:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 1 -- 172.21.15.64:0/100775312 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fda70001bb0 con 0x7fda78135f70 2024-09-17T15:50:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:50:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 10 monclient: dump: 2024-09-17T15:50:03.391 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:03.392 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:03.393 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda7eacd640 5 monclient: authenticate success, global_id 5995 2024-09-17T15:50:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda7eacd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 1 -- 172.21.15.64:0/100775312 <== mon.1 v1:172.21.15.100:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda70001dd0 con 0x7fda78135f70 2024-09-17T15:50:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda7eacd640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda7eacd640 4 set_mon_vals no callback set 2024-09-17T15:50:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 1 -- 172.21.15.64:0/100775312 <== mon.1 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fda700020c0 con 0x7fda78135f70 2024-09-17T15:50:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda777fe640 10 monclient: dump: 2024-09-17T15:50:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:03.393 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:03.394 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:03.395 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda7eacd640 10 monclient: shutdown 2024-09-17T15:50:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda7eacd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda7eacd640 1 -- 172.21.15.64:0/100775312 >> v1:172.21.15.100:6789/0 conn(0x7fda78135f70 legacy=0x7fda781415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda7eacd640 1 -- 172.21.15.64:0/100775312 shutdown_connections 2024-09-17T15:50:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda7eacd640 1 -- 172.21.15.64:0/100775312 >> 172.21.15.64:0/100775312 conn(0x7fda7806db30 msgr2=0x7fda780ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:03.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda7eacd640 1 -- 172.21.15.64:0/100775312 shutdown_connections 2024-09-17T15:50:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.381+0000 7fda7eacd640 1 -- 172.21.15.64:0/100775312 wait complete. 2024-09-17T15:50:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.382+0000 7fda7eacd640 10 monclient: build_initial_monmap 2024-09-17T15:50:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.382+0000 7fda7eacd640 10 monclient: monmap: 2024-09-17T15:50:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:03.396 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.382+0000 7fda7eacd640 1 Processor -- start 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.382+0000 7fda7eacd640 1 -- start start 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.382+0000 7fda7eacd640 10 monclient: init 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda7eacd640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda7eacd640 10 monclient: _add_conns ranks=[6,8,1,2,7,4,5,3,0] 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda7eacd640 10 monclient(hunting): picked mon.noname-d con 0x7fda780a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda7eacd640 10 monclient(hunting): picked mon.noname-e con 0x7fda78135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:50:03.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda7eacd640 10 monclient(hunting): picked mon.noname-f con 0x7fda78135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda7eacd640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda780a0b10 con 0x7fda78135f70 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda7eacd640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda780a92f0 con 0x7fda780a5520 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda77fff640 1 -- >> v1:172.21.15.64:6793/0 conn(0x7fda78135b50 legacy=0x7fda7809f2e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda7eacd640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda780a26c0 con 0x7fda78135b50 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda7eacd640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda7eacd640 10 monclient(hunting): authenticate will time out at 1001.058389s 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda7c869640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fda780a5520 0x7fda7809ebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:44092/0 (socket says 172.21.15.64:44092) 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda7c869640 1 -- 172.21.15.64:0/676539794 learned_addr learned my addr 172.21.15.64:0/676539794 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030603429 0 0) 0x7fda6c003a80 con 0x7fda780a5520 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 10 monclient(hunting): my global_id is 124242 2024-09-17T15:50:03.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda5c003520 con 0x7fda780a5520 2024-09-17T15:50:03.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690882969 0 0) 0x7fda780a0b10 con 0x7fda78135f70 2024-09-17T15:50:03.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:03.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:03.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:03.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 10 monclient(hunting): my global_id is 6004 2024-09-17T15:50:03.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda6c003a80 con 0x7fda78135f70 2024-09-17T15:50:03.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2051536599 0 0) 0x7fda5c003520 con 0x7fda780a5520 2024-09-17T15:50:03.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda780a0b10 con 0x7fda780a5520 2024-09-17T15:50:03.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fda6c004570 con 0x7fda780a5520 2024-09-17T15:50:03.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:03.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:50:03.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 10 monclient(hunting): dump: 2024-09-17T15:50:03.399 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:03.400 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:03.401 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.383+0000 7fda757fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2696556044 0 0) 0x7fda780a0b10 con 0x7fda780a5520 2024-09-17T15:50:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 >> v1:172.21.15.64:6793/0 conn(0x7fda78135b50 legacy=0x7fda7809f2e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 >> v1:172.21.15.100:6789/0 conn(0x7fda78135f70 legacy=0x7fda781d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 10 monclient: _finish_hunting 0 2024-09-17T15:50:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 1 monclient: found mon.d 2024-09-17T15:50:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:50:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda78134f20 con 0x7fda780a5520 2024-09-17T15:50:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 10 monclient: _finish_auth 0 2024-09-17T15:50:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 10 monclient: _check_auth_tickets 2024-09-17T15:50:03.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:03.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda7eacd640 5 monclient: authenticate success, global_id 124242 2024-09-17T15:50:03.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda7eacd640 10 monclient: _renew_subs 2024-09-17T15:50:03.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda7eacd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:50:03.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda7eacd640 1 -- 172.21.15.64:0/676539794 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fda780a11a0 con 0x7fda780a5520 2024-09-17T15:50:03.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda7eacd640 10 monclient: _renew_subs 2024-09-17T15:50:03.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda7eacd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:50:03.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda7eacd640 1 -- 172.21.15.64:0/676539794 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fda780a29a0 con 0x7fda780a5520 2024-09-17T15:50:03.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda6c004790 con 0x7fda780a5520 2024-09-17T15:50:03.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:03.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda76ffd640 4 set_mon_vals no callback set 2024-09-17T15:50:03.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fda6c004d80 con 0x7fda780a5520 2024-09-17T15:50:03.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:03.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:50:03.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.384+0000 7fda757fa640 10 monclient: dump: 2024-09-17T15:50:03.406 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:03.406 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:03.406 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.406 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:03.406 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:03.406 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:03.406 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:03.406 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:03.406 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:03.406 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:03.406 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:03.407 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:03.407 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:03.407 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:03.407 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:03.407 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:03.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.385+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fda6c004fe0 con 0x7fda780a5520 2024-09-17T15:50:03.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.385+0000 7fda7eacd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:03.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.385+0000 7fda7eacd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:03.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.385+0000 7fda7eacd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:50:03.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.385+0000 7fda7eacd640 1 -- 172.21.15.64:0/676539794 --> v1:172.21.15.64:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda58005080 con 0x7fda780a5520 2024-09-17T15:50:03.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.385+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(197..197 src has 1..197) ==== 3898+0+0 (unknown 1479433347 0 0) 0x7fda6c04ed20 con 0x7fda780a5520 2024-09-17T15:50:03.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.385+0000 7fda77fff640 10 monclient: get_auth_request con 0x7fda5c0400e0 auth_method 0 2024-09-17T15:50:03.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.390+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 <== mon.6 v1:172.21.15.64:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fda780a92f0 con 0x7fda780a5520 2024-09-17T15:50:03.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.390+0000 7fda757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:03.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.390+0000 7fda757fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:03.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.644+0000 7fda7eacd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:50:03.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.644+0000 7fda7eacd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:03.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.644+0000 7fda7eacd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:50:03.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.644+0000 7fda7eacd640 1 -- 172.21.15.64:0/676539794 --> v1:172.21.15.64:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fda58005370 con 0x7fda780a5520 2024-09-17T15:50:03.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.644+0000 7fda757fa640 1 -- 172.21.15.64:0/676539794 <== mon.6 v1:172.21.15.64:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 1273516324) 0x7fda6c0179f0 con 0x7fda780a5520 2024-09-17T15:50:03.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.644+0000 7fda757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:03.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.644+0000 7fda757fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:03.646 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":24,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:50:03.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.646+0000 7fda7eacd640 1 -- 172.21.15.64:0/676539794 >> v1:172.21.15.100:6812/4210963484 conn(0x7fda5c0400e0 legacy=0x7fda5c0425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:03.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.646+0000 7fda7eacd640 10 monclient: shutdown 2024-09-17T15:50:03.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.646+0000 7fda7eacd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:03.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.646+0000 7fda7eacd640 1 -- 172.21.15.64:0/676539794 >> v1:172.21.15.64:6792/0 conn(0x7fda780a5520 legacy=0x7fda7809ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:03.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.646+0000 7fda7eacd640 1 -- 172.21.15.64:0/676539794 shutdown_connections 2024-09-17T15:50:03.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.646+0000 7fda7eacd640 1 -- 172.21.15.64:0/676539794 >> 172.21.15.64:0/676539794 conn(0x7fda7806db30 msgr2=0x7fda780ab7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:03.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.646+0000 7fda7eacd640 1 -- 172.21.15.64:0/676539794 shutdown_connections 2024-09-17T15:50:03.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:03.646+0000 7fda7eacd640 1 -- 172.21.15.64:0/676539794 wait complete. 2024-09-17T15:50:06.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:50:06.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.771+0000 7f1afc2fe640 10 monclient: get_monmap_and_config 2024-09-17T15:50:06.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.771+0000 7f1afc2fe640 10 monclient: build_initial_monmap 2024-09-17T15:50:06.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.772+0000 7f1afc2fe640 10 monclient: monmap: 2024-09-17T15:50:06.774 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:06.774 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:06.774 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:06.772873+0000 2024-09-17T15:50:06.774 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:06.772873+0000 2024-09-17T15:50:06.774 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:06.774 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:06.774 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:06.774 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:06.774 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:06.775 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:06.775 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:06.775 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:06.775 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:06.775 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:06.775 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:06.775 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:06.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.772+0000 7f1afc2fe640 1 Processor -- start 2024-09-17T15:50:06.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.772+0000 7f1afc2fe640 1 -- start start 2024-09-17T15:50:06.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.772+0000 7f1afc2fe640 10 monclient: init 2024-09-17T15:50:06.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1afc2fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:06.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1afc2fe640 10 monclient: _add_conns ranks=[3,4,1,0,8,5,2,7,6] 2024-09-17T15:50:06.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1afc2fe640 10 monclient(hunting): picked mon.noname-g con 0x7f1af40a5520 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:06.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1afc2fe640 10 monclient(hunting): picked mon.noname-c con 0x7f1af4135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:06.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1afc2fe640 10 monclient(hunting): picked mon.noname-f con 0x7f1af4135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:06.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1afc2fe640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1af409d6f0 con 0x7f1af4135f70 2024-09-17T15:50:06.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1afc2fe640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1af409d9d0 con 0x7f1af40a5520 2024-09-17T15:50:06.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1afc2fe640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1af409dcb0 con 0x7f1af4135b50 2024-09-17T15:50:06.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1afc2fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:06.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1afc2fe640 10 monclient(hunting): authenticate will time out at 1004.448539s 2024-09-17T15:50:06.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1af9899640 1 -- >> v1:172.21.15.64:6791/0 conn(0x7f1af4135b50 legacy=0x7f1af413deb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:06.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1afa09a640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f1af40a5520 0x7f1af40a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:50500/0 (socket says 172.21.15.64:50500) 2024-09-17T15:50:06.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1afa89b640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f1af4135f70 0x7f1af41415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.64:33006/0 (socket says 172.21.15.64:33006) 2024-09-17T15:50:06.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.773+0000 7f1afa09a640 1 -- 172.21.15.64:0/57284849 learned_addr learned my addr 172.21.15.64:0/57284849 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:06.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 1 -- 172.21.15.64:0/57284849 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1696970022 0 0) 0x7f1af409d6f0 con 0x7f1af4135f70 2024-09-17T15:50:06.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient(hunting): _negotiate 2024-09-17T15:50:06.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:06.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:06.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient(hunting): my global_id is 6013 2024-09-17T15:50:06.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 1 -- 172.21.15.64:0/57284849 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ae0003520 con 0x7f1af4135f70 2024-09-17T15:50:06.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 1 -- 172.21.15.64:0/57284849 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2254677020 0 0) 0x7f1ae0003520 con 0x7f1af4135f70 2024-09-17T15:50:06.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 1 -- 172.21.15.64:0/57284849 >> v1:172.21.15.64:6791/0 conn(0x7f1af4135b50 legacy=0x7f1af413deb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:06.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 1 -- 172.21.15.64:0/57284849 >> v1:172.21.15.100:6790/0 conn(0x7f1af40a5520 legacy=0x7f1af40a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:06.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient: _finish_hunting 0 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 1 monclient: found mon.noname-f 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.100:6789/0 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 1 -- 172.21.15.64:0/57284849 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af409ccd0 con 0x7f1af4135f70 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient: _finish_auth 0 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient: _check_auth_tickets 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 1 -- 172.21.15.64:0/57284849 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1aec001bb0 con 0x7f1af4135f70 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1afc2fe640 5 monclient: authenticate success, global_id 6013 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1afc2fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient: dump: 2024-09-17T15:50:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:06.783 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:06.784 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:06.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1afc2fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:06.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 1 -- 172.21.15.64:0/57284849 <== mon.1 v1:172.21.15.100:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1aec001dd0 con 0x7f1af4135f70 2024-09-17T15:50:06.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:06.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1afc2fe640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:06.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1afc2fe640 4 set_mon_vals no callback set 2024-09-17T15:50:06.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 1 -- 172.21.15.64:0/57284849 <== mon.1 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1aec0020c0 con 0x7f1af4135f70 2024-09-17T15:50:06.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:06.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:06.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.774+0000 7f1af9098640 10 monclient: dump: 2024-09-17T15:50:06.784 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:06.784 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:06.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.775+0000 7f1afc2fe640 10 monclient: shutdown 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.775+0000 7f1afc2fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.775+0000 7f1afc2fe640 1 -- 172.21.15.64:0/57284849 >> v1:172.21.15.100:6789/0 conn(0x7f1af4135f70 legacy=0x7f1af41415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.775+0000 7f1afc2fe640 1 -- 172.21.15.64:0/57284849 shutdown_connections 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.775+0000 7f1afc2fe640 1 -- 172.21.15.64:0/57284849 >> 172.21.15.64:0/57284849 conn(0x7f1af406db30 msgr2=0x7f1af40ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.775+0000 7f1afc2fe640 1 -- 172.21.15.64:0/57284849 shutdown_connections 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.775+0000 7f1afc2fe640 1 -- 172.21.15.64:0/57284849 wait complete. 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.775+0000 7f1afc2fe640 10 monclient: build_initial_monmap 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.775+0000 7f1afc2fe640 10 monclient: monmap: 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:06.786 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:06.787 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:06.787 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:06.787 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:06.787 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:06.787 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:06.787 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:06.787 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:06.787 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:06.787 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:06.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afc2fe640 1 Processor -- start 2024-09-17T15:50:06.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afc2fe640 1 -- start start 2024-09-17T15:50:06.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afc2fe640 10 monclient: init 2024-09-17T15:50:06.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afc2fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:06.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afc2fe640 10 monclient: _add_conns ranks=[3,4,6,7,5,2,1,0,8] 2024-09-17T15:50:06.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afc2fe640 10 monclient(hunting): picked mon.noname-g con 0x7f1af40a5520 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:06.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afc2fe640 10 monclient(hunting): picked mon.noname-c con 0x7f1af4135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:06.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afc2fe640 10 monclient(hunting): picked mon.noname-d con 0x7f1af4135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:50:06.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afc2fe640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1af40a0770 con 0x7f1af40a5520 2024-09-17T15:50:06.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afc2fe640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1af4134960 con 0x7f1af4135b50 2024-09-17T15:50:06.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afc2fe640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1af409e8b0 con 0x7f1af4135f70 2024-09-17T15:50:06.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afc2fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:06.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afc2fe640 10 monclient(hunting): authenticate will time out at 1004.451695s 2024-09-17T15:50:06.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1af9899640 1 -- >> v1:172.21.15.64:6791/0 conn(0x7f1af4135b50 legacy=0x7f1af4131f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:06.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afa89b640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f1af4135f70 0x7f1af4132630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:53014/0 (socket says 172.21.15.64:53014) 2024-09-17T15:50:06.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.776+0000 7f1afa89b640 1 -- 172.21.15.64:0/2211765361 learned_addr learned my addr 172.21.15.64:0/2211765361 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:06.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:06.977+0000 7f1af9899640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6791/0 conn(0x7f1af4135b50 legacy=0x7f1af4131f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:07.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:07.378+0000 7f1af9899640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6791/0 conn(0x7f1af4135b50 legacy=0x7f1af4131f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:08.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:08.179+0000 7f1af9899640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6791/0 conn(0x7f1af4135b50 legacy=0x7f1af4131f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:09.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 10 monclient(hunting): tick 2024-09-17T15:50:09.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:50:09.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 1 monclient(hunting): continuing hunt 2024-09-17T15:50:09.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:50:09.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6792/0 conn(0x7f1af4135f70 legacy=0x7f1af4132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6791/0 conn(0x7f1af4135b50 legacy=0x7f1af4131f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:09.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6790/0 conn(0x7f1af40a5520 legacy=0x7f1af4131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 10 monclient: _add_conns ranks=[8,5,2,0,7,1] 2024-09-17T15:50:09.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 10 monclient(hunting): picked mon.noname-e con 0x7f1adc0037c0 addr v1:172.21.15.64:6793/0 2024-09-17T15:50:09.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 10 monclient(hunting): picked mon.noname-h con 0x7f1adc007450 addr v1:172.21.15.100:6791/0 2024-09-17T15:50:09.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f1adc00b1e0 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:09.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1af4135340 con 0x7f1adc00b1e0 2024-09-17T15:50:09.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1af4134960 con 0x7f1adc007450 2024-09-17T15:50:09.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1adc011010 con 0x7f1adc0037c0 2024-09-17T15:50:09.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1ae9ffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:09.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1afa09a640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6793/0 conn(0x7f1adc0037c0 legacy=0x7f1adc005c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:50:09.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.777+0000 7f1af9899640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6791/0 conn(0x7f1adc007450 legacy=0x7f1adc009910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:50:09.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.978+0000 7f1afa09a640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6793/0 conn(0x7f1adc0037c0 legacy=0x7f1adc005c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:50:09.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:09.978+0000 7f1af9899640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6791/0 conn(0x7f1adc007450 legacy=0x7f1adc009910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:50:10.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:10.379+0000 7f1afa09a640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6793/0 conn(0x7f1adc0037c0 legacy=0x7f1adc005c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:50:10.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:10.379+0000 7f1af9899640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6791/0 conn(0x7f1adc007450 legacy=0x7f1adc009910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:50:11.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:11.180+0000 7f1afa09a640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6793/0 conn(0x7f1adc0037c0 legacy=0x7f1adc005c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:50:11.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:11.180+0000 7f1af9899640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6791/0 conn(0x7f1adc007450 legacy=0x7f1adc009910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:50:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 10 monclient(hunting): tick 2024-09-17T15:50:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:50:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 1 monclient(hunting): continuing hunt 2024-09-17T15:50:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:50:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6793/0 conn(0x7f1adc0037c0 legacy=0x7f1adc005c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6791/0 conn(0x7f1adc007450 legacy=0x7f1adc009910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:12.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6790/0 conn(0x7f1adc00b1e0 legacy=0x7f1adc00d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:12.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 10 monclient: _add_conns ranks=[0,1,7] 2024-09-17T15:50:12.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1afa89b640 1 -- 172.21.15.64:0/2211765361 reap_dead start 2024-09-17T15:50:12.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f1adc011320 addr v1:172.21.15.64:6789/0 2024-09-17T15:50:12.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 10 monclient(hunting): picked mon.noname-f con 0x7f1adc00ef30 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:12.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 10 monclient(hunting): picked mon.noname-i con 0x7f1adc00fa50 addr v1:172.21.15.100:6792/0 2024-09-17T15:50:12.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1adc0122b0 con 0x7f1adc011320 2024-09-17T15:50:12.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1af4134960 con 0x7f1adc00ef30 2024-09-17T15:50:12.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1adc011010 con 0x7f1adc00fa50 2024-09-17T15:50:12.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1ae9ffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:12.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1afa89b640 1 -- 172.21.15.64:0/2211765361 reap_dead start 2024-09-17T15:50:12.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.777+0000 7f1afa89b640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6792/0 conn(0x7f1adc00fa50 legacy=0x7f1adc019620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:50:12.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:12.978+0000 7f1afa89b640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6792/0 conn(0x7f1adc00fa50 legacy=0x7f1adc019620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:50:13.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:13.379+0000 7f1afa89b640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6792/0 conn(0x7f1adc00fa50 legacy=0x7f1adc019620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:50:14.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:14.180+0000 7f1afa89b640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6792/0 conn(0x7f1adc00fa50 legacy=0x7f1adc019620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:50:15.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 10 monclient(hunting): tick 2024-09-17T15:50:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:50:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 1 monclient(hunting): continuing hunt 2024-09-17T15:50:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:50:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6792/0 conn(0x7f1adc00fa50 legacy=0x7f1adc019620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6789/0 conn(0x7f1adc00ef30 legacy=0x7f1adc00f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6789/0 conn(0x7f1adc011320 legacy=0x7f1adc011700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 10 monclient: _add_conns ranks=[4,7,0,1,5,2,6,3,8] 2024-09-17T15:50:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f1adc00a350 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 10 monclient(hunting): picked mon.noname-i con 0x7f1adc00fa50 addr v1:172.21.15.100:6792/0 2024-09-17T15:50:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f1adc011320 addr v1:172.21.15.64:6789/0 2024-09-17T15:50:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1adc00a0e0 con 0x7f1adc011320 2024-09-17T15:50:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1adc011010 con 0x7f1adc00a350 2024-09-17T15:50:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1adc009920 con 0x7f1adc00fa50 2024-09-17T15:50:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1afa09a640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6791/0 conn(0x7f1adc00a350 legacy=0x7f1adc005c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1ae9ffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.778+0000 7f1af9899640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6792/0 conn(0x7f1adc00fa50 legacy=0x7f1adc00d680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:50:15.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.979+0000 7f1afa09a640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6791/0 conn(0x7f1adc00a350 legacy=0x7f1adc005c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:15.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:15.979+0000 7f1af9899640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6792/0 conn(0x7f1adc00fa50 legacy=0x7f1adc00d680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:50:16.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:16.380+0000 7f1afa09a640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6791/0 conn(0x7f1adc00a350 legacy=0x7f1adc005c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:16.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:16.380+0000 7f1af9899640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6792/0 conn(0x7f1adc00fa50 legacy=0x7f1adc00d680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:50:17.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.181+0000 7f1afa09a640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6791/0 conn(0x7f1adc00a350 legacy=0x7f1adc005c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:17.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.181+0000 7f1af9899640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6792/0 conn(0x7f1adc00fa50 legacy=0x7f1adc00d680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:50:17.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.482+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3297329261 0 0) 0x7f1adc00a0e0 con 0x7f1adc011320 2024-09-17T15:50:17.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.482+0000 7f1aeaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:17.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.482+0000 7f1aeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:17.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.482+0000 7f1aeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:17.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.482+0000 7f1aeaffd640 10 monclient(hunting): my global_id is 124218 2024-09-17T15:50:17.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.482+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ad8003520 con 0x7f1adc011320 2024-09-17T15:50:17.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.489+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1723660910 0 0) 0x7f1ad8003520 con 0x7f1adc011320 2024-09-17T15:50:17.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.489+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1adc00a0e0 con 0x7f1adc011320 2024-09-17T15:50:17.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.489+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1aec001f80 con 0x7f1adc011320 2024-09-17T15:50:17.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.489+0000 7f1aeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:17.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.489+0000 7f1aeaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:17.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.489+0000 7f1aeaffd640 10 monclient(hunting): dump: 2024-09-17T15:50:17.492 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:17.492 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:17.493 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.489+0000 7f1aeaffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.489+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3155807188 0 0) 0x7f1adc00a0e0 con 0x7f1adc011320 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6792/0 conn(0x7f1adc00fa50 legacy=0x7f1adc00d680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6791/0 conn(0x7f1adc00a350 legacy=0x7f1adc005c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 10 monclient: _finish_hunting 0 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 1 monclient: found mon.a 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1adc009b90 con 0x7f1adc011320 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 10 monclient: _finish_auth 0 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1afc2fe640 5 monclient: authenticate success, global_id 124218 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1afc2fe640 10 monclient: _renew_subs 2024-09-17T15:50:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1afc2fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1afc2fe640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1af40a0490 con 0x7f1adc011320 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1afc2fe640 10 monclient: _renew_subs 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1afc2fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1afc2fe640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1af4134e20 con 0x7f1adc011320 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1aec001b90 con 0x7f1adc011320 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f1aec005490 con 0x7f1adc011320 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1af8897640 4 set_mon_vals no callback set 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.490+0000 7f1aeaffd640 10 monclient: dump: 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.496 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:17.496 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:17.496 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:17.496 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:17.496 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:17.496 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:17.496 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:17.496 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:17.496 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:17.496 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:17.496 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:17.496 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:17.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.491+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f1aec011d70 con 0x7f1adc011320 2024-09-17T15:50:17.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.491+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(197..197 src has 1..197) ==== 3898+0+0 (unknown 1479433347 0 0) 0x7f1aec04ee60 con 0x7f1adc011320 2024-09-17T15:50:17.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.491+0000 7f1afc2fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:17.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.491+0000 7f1afc2fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.491+0000 7f1afc2fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:50:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.491+0000 7f1afc2fe640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad0005080 con 0x7f1adc011320 2024-09-17T15:50:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.491+0000 7f1afa09a640 10 monclient: get_auth_request con 0x7f1ad803ffd0 auth_method 0 2024-09-17T15:50:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.496+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f1aec017af0 con 0x7f1adc011320 2024-09-17T15:50:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.496+0000 7f1aeaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.496+0000 7f1aeaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:17.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.761+0000 7f1afc2fe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:50:17.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.761+0000 7f1afc2fe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:17.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.761+0000 7f1afc2fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:50:17.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.761+0000 7f1afc2fe640 1 -- 172.21.15.64:0/2211765361 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1ad0005370 con 0x7f1adc011320 2024-09-17T15:50:17.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.761+0000 7f1aeaffd640 1 -- 172.21.15.64:0/2211765361 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2705 (unknown 254746657 0 1023747192) 0x7f1aec0173f0 con 0x7f1adc011320 2024-09-17T15:50:17.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.761+0000 7f1aeaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:17.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.761+0000 7f1aeaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:17.764 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":28,"quorum":[0,1,2,3,6],"quorum_names":["a","f","b","g","d"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:50:17.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.764+0000 7f1afc2fe640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.100:6812/4210963484 conn(0x7f1ad803ffd0 legacy=0x7f1ad8042490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:17.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.764+0000 7f1afc2fe640 10 monclient: shutdown 2024-09-17T15:50:17.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.764+0000 7f1afc2fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:17.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.764+0000 7f1afc2fe640 1 -- 172.21.15.64:0/2211765361 >> v1:172.21.15.64:6789/0 conn(0x7f1adc011320 legacy=0x7f1adc011700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:17.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.764+0000 7f1afa89b640 1 -- 172.21.15.64:0/2211765361 reap_dead start 2024-09-17T15:50:17.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.764+0000 7f1afc2fe640 1 -- 172.21.15.64:0/2211765361 shutdown_connections 2024-09-17T15:50:17.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.764+0000 7f1afc2fe640 1 -- 172.21.15.64:0/2211765361 >> 172.21.15.64:0/2211765361 conn(0x7f1af406db30 msgr2=0x7f1af41334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:17.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.764+0000 7f1afc2fe640 1 -- 172.21.15.64:0/2211765361 shutdown_connections 2024-09-17T15:50:17.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.764+0000 7f1afc2fe640 1 -- 172.21.15.64:0/2211765361 wait complete. 2024-09-17T15:50:17.776 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-09-17T15:50:17.777 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon a 2024-09-17T15:50:17.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:50:17.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.876+0000 7f467a8b1640 10 monclient: get_monmap_and_config 2024-09-17T15:50:17.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.876+0000 7f467a8b1640 10 monclient: build_initial_monmap 2024-09-17T15:50:17.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.877+0000 7f467a8b1640 10 monclient: monmap: 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:17.877854+0000 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:17.877854+0000 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:17.880 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:17.880 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:17.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.877+0000 7f467a8b1640 1 Processor -- start 2024-09-17T15:50:17.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.877+0000 7f467a8b1640 1 -- start start 2024-09-17T15:50:17.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.877+0000 7f467a8b1640 10 monclient: init 2024-09-17T15:50:17.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f467a8b1640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:17.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f467a8b1640 10 monclient: _add_conns ranks=[6,4,1,5,2,3,0,7,8] 2024-09-17T15:50:17.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f467a8b1640 10 monclient(hunting): picked mon.noname-d con 0x7f46740a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:50:17.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f467a8b1640 10 monclient(hunting): picked mon.noname-c con 0x7f4674135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:17.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f467a8b1640 10 monclient(hunting): picked mon.noname-f con 0x7f4674135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:17.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f467a8b1640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467409d6f0 con 0x7f4674135f70 2024-09-17T15:50:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f467a8b1640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467409d9d0 con 0x7f4674135b50 2024-09-17T15:50:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f467a8b1640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467409dcb0 con 0x7f46740a5520 2024-09-17T15:50:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f467a8b1640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f46737fe640 1 -- >> v1:172.21.15.64:6791/0 conn(0x7f4674135b50 legacy=0x7f467413deb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f467a8b1640 10 monclient(hunting): authenticate will time out at 1015.553460s 2024-09-17T15:50:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4673fff640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f46740a5520 0x7f46740a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:51992/0 (socket says 172.21.15.64:51992) 2024-09-17T15:50:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4673fff640 1 -- 172.21.15.64:0/640818513 learned_addr learned my addr 172.21.15.64:0/640818513 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 1 -- 172.21.15.64:0/640818513 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2673686750 0 0) 0x7f467409dcb0 con 0x7f46740a5520 2024-09-17T15:50:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 10 monclient(hunting): my global_id is 124251 2024-09-17T15:50:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 1 -- 172.21.15.64:0/640818513 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4660003520 con 0x7f46740a5520 2024-09-17T15:50:17.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 1 -- 172.21.15.64:0/640818513 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1286272043 0 0) 0x7f467409d6f0 con 0x7f4674135f70 2024-09-17T15:50:17.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:17.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:17.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:17.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 10 monclient(hunting): my global_id is 6022 2024-09-17T15:50:17.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 1 -- 172.21.15.64:0/640818513 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f467409dcb0 con 0x7f4674135f70 2024-09-17T15:50:17.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 1 -- 172.21.15.64:0/640818513 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 703925044 0 0) 0x7f4660003520 con 0x7f46740a5520 2024-09-17T15:50:17.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.878+0000 7f4672ffd640 1 -- 172.21.15.64:0/640818513 >> v1:172.21.15.64:6791/0 conn(0x7f4674135b50 legacy=0x7f467413deb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:17.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 1 -- 172.21.15.64:0/640818513 >> v1:172.21.15.100:6789/0 conn(0x7f4674135f70 legacy=0x7f46741415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:17.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:50:17.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 1 monclient: found mon.noname-d 2024-09-17T15:50:17.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 1 -- 172.21.15.64:0/640818513 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467409ccd0 con 0x7f46740a5520 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 10 monclient: _finish_auth 0 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f467a8b1640 5 monclient: authenticate success, global_id 124251 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f467a8b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 1 -- 172.21.15.64:0/640818513 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4664001830 con 0x7f46740a5520 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 10 monclient: dump: 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:17.891 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.892 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.892 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:17.892 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:17.892 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:17.892 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:17.892 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:17.892 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:17.892 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:17.892 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:17.892 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:17.892 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:17.892 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:17.892 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:17.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f467a8b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:17.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 1 -- 172.21.15.64:0/640818513 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f46640019d0 con 0x7f46740a5520 2024-09-17T15:50:17.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:17.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f467a8b1640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:17.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 1 -- 172.21.15.64:0/640818513 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4664004be0 con 0x7f46740a5520 2024-09-17T15:50:17.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f467a8b1640 4 set_mon_vals no callback set 2024-09-17T15:50:17.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:17.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:50:17.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f4672ffd640 10 monclient: dump: 2024-09-17T15:50:17.893 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:17.893 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:17.893 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.893 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f467a8b1640 10 monclient: shutdown 2024-09-17T15:50:17.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f467a8b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:17.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f467a8b1640 1 -- 172.21.15.64:0/640818513 >> v1:172.21.15.64:6792/0 conn(0x7f46740a5520 legacy=0x7f46740a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:17.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f467a8b1640 1 -- 172.21.15.64:0/640818513 shutdown_connections 2024-09-17T15:50:17.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f467a8b1640 1 -- 172.21.15.64:0/640818513 >> 172.21.15.64:0/640818513 conn(0x7f467406db30 msgr2=0x7f46740ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:17.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f467a8b1640 1 -- 172.21.15.64:0/640818513 shutdown_connections 2024-09-17T15:50:17.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.879+0000 7f467a8b1640 1 -- 172.21.15.64:0/640818513 wait complete. 2024-09-17T15:50:17.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.880+0000 7f467a8b1640 10 monclient: build_initial_monmap 2024-09-17T15:50:17.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.880+0000 7f467a8b1640 10 monclient: monmap: 2024-09-17T15:50:17.895 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:17.895 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:17.895 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:17.895 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:17.895 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:17.895 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:17.896 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:17.896 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:17.896 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:17.896 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:17.896 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:17.896 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:17.896 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:17.896 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:17.896 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:17.896 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:17.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.880+0000 7f467a8b1640 1 Processor -- start 2024-09-17T15:50:17.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.880+0000 7f467a8b1640 1 -- start start 2024-09-17T15:50:17.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.880+0000 7f467a8b1640 10 monclient: init 2024-09-17T15:50:17.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f467a8b1640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:17.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f467a8b1640 10 monclient: _add_conns ranks=[4,2,7,3,0,1,8,5,6] 2024-09-17T15:50:17.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f467a8b1640 10 monclient(hunting): picked mon.noname-c con 0x7f46740a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:17.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f467a8b1640 10 monclient(hunting): picked mon.noname-b con 0x7f4674135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:17.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f467a8b1640 10 monclient(hunting): picked mon.noname-i con 0x7f4674135f70 addr v1:172.21.15.100:6792/0 2024-09-17T15:50:17.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f467a8b1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467409e980 con 0x7f4674135b50 2024-09-17T15:50:17.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4673fff640 1 -- >> v1:172.21.15.64:6791/0 conn(0x7f46740a5520 legacy=0x7f46740a49f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:17.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f467a8b1640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46740a02b0 con 0x7f46740a5520 2024-09-17T15:50:17.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f467a8b1640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467409ef20 con 0x7f4674135f70 2024-09-17T15:50:17.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f467a8b1640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:17.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f467a8b1640 10 monclient(hunting): authenticate will time out at 1015.556419s 2024-09-17T15:50:17.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f46737fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f4674135b50 0x7f46741d2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:34486/0 (socket says 172.21.15.64:34486) 2024-09-17T15:50:17.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f46737fe640 1 -- 172.21.15.64:0/1015107899 learned_addr learned my addr 172.21.15.64:0/1015107899 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:17.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4678e4e640 1 -- 172.21.15.64:0/1015107899 >> v1:172.21.15.100:6792/0 conn(0x7f4674135f70 legacy=0x7f46741d6140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:50:17.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3126960647 0 0) 0x7f467409e980 con 0x7f4674135b50 2024-09-17T15:50:17.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4670ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:50:17.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4670ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:17.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4670ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:17.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4670ff9640 10 monclient(hunting): my global_id is 124283 2024-09-17T15:50:17.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4654003520 con 0x7f4674135b50 2024-09-17T15:50:17.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3877834717 0 0) 0x7f4654003520 con 0x7f4674135b50 2024-09-17T15:50:17.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f467409e980 con 0x7f4674135b50 2024-09-17T15:50:17.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f465c0043f0 con 0x7f4674135b50 2024-09-17T15:50:17.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4670ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:17.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4670ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4670ff9640 10 monclient(hunting): dump: 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:17.899 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:17.900 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:17.900 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:17.900 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:17.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.881+0000 7f4670ff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:17.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1693698375 0 0) 0x7f467409e980 con 0x7f4674135b50 2024-09-17T15:50:17.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 >> v1:172.21.15.100:6792/0 conn(0x7f4674135f70 legacy=0x7f46741d6140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:17.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 >> v1:172.21.15.64:6791/0 conn(0x7f46740a5520 legacy=0x7f46740a49f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:17.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 10 monclient: _finish_hunting 0 2024-09-17T15:50:17.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 1 monclient: found mon.b 2024-09-17T15:50:17.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:17.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:17.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467409ffd0 con 0x7f4674135b50 2024-09-17T15:50:17.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 10 monclient: _finish_auth 0 2024-09-17T15:50:17.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:50:17.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:17.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f467a8b1640 5 monclient: authenticate success, global_id 124283 2024-09-17T15:50:17.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f467a8b1640 10 monclient: _renew_subs 2024-09-17T15:50:17.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f467a8b1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:17.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f467a8b1640 1 -- 172.21.15.64:0/1015107899 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f46740a5100 con 0x7f4674135b50 2024-09-17T15:50:17.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f467a8b1640 10 monclient: _renew_subs 2024-09-17T15:50:17.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f467a8b1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:17.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f467a8b1640 1 -- 172.21.15.64:0/1015107899 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f46741ca310 con 0x7f4674135b50 2024-09-17T15:50:17.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f465c004610 con 0x7f4674135b50 2024-09-17T15:50:17.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:17.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f465c004bc0 con 0x7f4674135b50 2024-09-17T15:50:17.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:17.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:17.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f46727fc640 4 set_mon_vals no callback set 2024-09-17T15:50:17.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 10 monclient: dump: 2024-09-17T15:50:17.902 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:17.902 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:17.902 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.902 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.902 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:17.902 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:17.902 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:17.902 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f467a8b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f467a8b1640 1 -- 172.21.15.64:0/1015107899 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f46580065c0 con 0x7f4658002860 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.882+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f465c004d60 con 0x7f4674135b50 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(197..197 src has 1..197) ==== 3898+0+0 (unknown 1479433347 0 0) 0x7f465c04ed60 con 0x7f4674135b50 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1221239353 0 0) 0x7f46580065c0 con 0x7f4658002860 2024-09-17T15:50:17.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:50:17.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:17.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:17.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 10 monclient(hunting): my global_id is 124227 2024-09-17T15:50:17.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f467409e980 con 0x7f4658002860 2024-09-17T15:50:17.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 597849606 0 0) 0x7f467409e980 con 0x7f4658002860 2024-09-17T15:50:17.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46580065c0 con 0x7f4658002860 2024-09-17T15:50:17.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4664004be0 con 0x7f4658002860 2024-09-17T15:50:17.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:17.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:17.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 10 monclient: dump: 2024-09-17T15:50:17.904 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:17.904 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:17.904 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.905 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.905 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:17.905 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:17.905 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:17.905 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:17.905 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:17.905 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:17.905 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:17.905 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:17.905 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:17.905 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:17.905 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:17.905 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:17.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4678e4e640 10 monclient: get_auth_request con 0x7f46540401f0 auth_method 0 2024-09-17T15:50:17.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2244008346 0 0) 0x7f46580065c0 con 0x7f4658002860 2024-09-17T15:50:17.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.883+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 --> v1:172.21.15.64:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f465800a160 con 0x7f4658002860 2024-09-17T15:50:17.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.884+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f46640014f0 con 0x7f4658002860 2024-09-17T15:50:17.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.884+0000 7f4670ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:17.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.884+0000 7f4670ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:17.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.884+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 >> v1:172.21.15.64:6789/0 conn(0x7f4658002860 legacy=0x7f4658004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:17.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.884+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 >> v1:172.21.15.64:6789/0 conn(0x7f4658002860 legacy=0x7f4658004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:17.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.890+0000 7f467a8b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:17.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.890+0000 7f467a8b1640 1 -- 172.21.15.64:0/1015107899 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4658009e50 con 0x7f4658009110 2024-09-17T15:50:17.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.890+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 760815639 0 0) 0x7f4658009e50 con 0x7f4658009110 2024-09-17T15:50:17.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.890+0000 7f4670ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:50:17.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.890+0000 7f4670ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:17.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.890+0000 7f4670ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:17.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.890+0000 7f4670ff9640 10 monclient(hunting): my global_id is 124236 2024-09-17T15:50:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.890+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46580065c0 con 0x7f4658009110 2024-09-17T15:50:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1598651696 0 0) 0x7f46580065c0 con 0x7f4658009110 2024-09-17T15:50:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4658009e50 con 0x7f4658009110 2024-09-17T15:50:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4664001910 con 0x7f4658009110 2024-09-17T15:50:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4670ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4670ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4670ff9640 10 monclient: dump: 2024-09-17T15:50:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:17.910 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:17.910 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:17.910 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:17.910 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:17.910 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:17.910 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:17.910 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:17.910 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:17.910 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:17.910 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:17.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 32778740 0 0) 0x7f4658009e50 con 0x7f4658009110 2024-09-17T15:50:17.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 --> v1:172.21.15.64:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f465800c3f0 con 0x7f4658009110 2024-09-17T15:50:17.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7666 (unknown 0 0 965260137) 0x7f4664005340 con 0x7f4658009110 2024-09-17T15:50:17.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4670ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:17.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4670ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:17.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 >> v1:172.21.15.64:6789/0 conn(0x7f4658009110 legacy=0x7f46580095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:17.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4670ff9640 1 -- 172.21.15.64:0/1015107899 >> v1:172.21.15.64:6789/0 conn(0x7f4658009110 legacy=0x7f46580095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:17.911 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:50:17.911 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 28, 2024-09-17T15:50:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 24809, 2024-09-17T15:50:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: 6 2024-09-17T15:50:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:50:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:17.912 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:17.913 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:17.914 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:17.915 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:17.916 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.919 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:17.920 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:17.921 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:17.922 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:17.923 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.924 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:17.925 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:17.926 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:17.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.929 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:17.930 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:50:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:50:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:17.931 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:50:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:17.932 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:17.932 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:17.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f467a8b1640 1 -- 172.21.15.64:0/1015107899 >> v1:172.21.15.100:6812/4210963484 conn(0x7f46540401f0 legacy=0x7f46540426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:17.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f467a8b1640 10 monclient: shutdown 2024-09-17T15:50:17.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f467a8b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:17.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f467a8b1640 1 -- 172.21.15.64:0/1015107899 >> v1:172.21.15.64:6790/0 conn(0x7f4674135b50 legacy=0x7f46741d2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:17.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4678e4e640 1 -- 172.21.15.64:0/1015107899 reap_dead start 2024-09-17T15:50:17.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f4678e4e640 1 -- 172.21.15.64:0/1015107899 reap_dead start 2024-09-17T15:50:17.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f467a8b1640 1 -- 172.21.15.64:0/1015107899 shutdown_connections 2024-09-17T15:50:17.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.891+0000 7f467a8b1640 1 -- 172.21.15.64:0/1015107899 >> 172.21.15.64:0/1015107899 conn(0x7f467406db30 msgr2=0x7f46741334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:17.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.892+0000 7f467a8b1640 1 -- 172.21.15.64:0/1015107899 shutdown_connections 2024-09-17T15:50:17.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:17.892+0000 7f467a8b1640 1 -- 172.21.15.64:0/1015107899 wait complete. 2024-09-17T15:50:17.936 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:50:17.937 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon b 2024-09-17T15:50:17.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:50:18.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.037+0000 7f8a6abd3640 10 monclient: get_monmap_and_config 2024-09-17T15:50:18.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.037+0000 7f8a6abd3640 10 monclient: build_initial_monmap 2024-09-17T15:50:18.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.037+0000 7f8a6abd3640 10 monclient: monmap: 2024-09-17T15:50:18.039 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:18.039 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.039 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:18.038409+0000 2024-09-17T15:50:18.039 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:18.038409+0000 2024-09-17T15:50:18.039 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:18.039 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.040 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:18.040 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:18.040 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:18.040 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:18.040 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:18.040 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:18.040 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:18.040 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:18.040 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:18.040 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.038+0000 7f8a6abd3640 1 Processor -- start 2024-09-17T15:50:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.038+0000 7f8a6abd3640 1 -- start start 2024-09-17T15:50:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.038+0000 7f8a6abd3640 10 monclient: init 2024-09-17T15:50:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.038+0000 7f8a6abd3640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.038+0000 7f8a6abd3640 10 monclient: _add_conns ranks=[4,2,6,7,3,0,5,1,8] 2024-09-17T15:50:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.038+0000 7f8a6abd3640 10 monclient(hunting): picked mon.noname-c con 0x7f8a640a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.038+0000 7f8a6abd3640 10 monclient(hunting): picked mon.noname-b con 0x7f8a64135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.038+0000 7f8a6abd3640 10 monclient(hunting): picked mon.noname-d con 0x7f8a64135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:50:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.038+0000 7f8a6896f640 1 -- >> v1:172.21.15.64:6791/0 conn(0x7f8a640a5520 legacy=0x7f8a640a5900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a6abd3640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a6409d6f0 con 0x7f8a64135b50 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a6abd3640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a6409d9d0 con 0x7f8a640a5520 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a6abd3640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a6409dcb0 con 0x7f8a64135f70 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a6abd3640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a6abd3640 10 monclient(hunting): authenticate will time out at 1015.714125s 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5bfff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f8a64135b50 0x7f8a6413deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:34500/0 (socket says 172.21.15.64:34500) 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5bfff640 1 -- 172.21.15.64:0/2101483619 learned_addr learned my addr 172.21.15.64:0/2101483619 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 1 -- 172.21.15.64:0/2101483619 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2978492278 0 0) 0x7f8a6409dcb0 con 0x7f8a64135f70 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient(hunting): my global_id is 124260 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 1 -- 172.21.15.64:0/2101483619 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a4c003520 con 0x7f8a64135f70 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 1 -- 172.21.15.64:0/2101483619 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195910381 0 0) 0x7f8a6409d6f0 con 0x7f8a64135b50 2024-09-17T15:50:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient(hunting): my global_id is 124292 2024-09-17T15:50:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 1 -- 172.21.15.64:0/2101483619 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a6409dcb0 con 0x7f8a64135b50 2024-09-17T15:50:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 1 -- 172.21.15.64:0/2101483619 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3860260219 0 0) 0x7f8a4c003520 con 0x7f8a64135f70 2024-09-17T15:50:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 1 -- 172.21.15.64:0/2101483619 >> v1:172.21.15.64:6791/0 conn(0x7f8a640a5520 legacy=0x7f8a640a5900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 1 -- 172.21.15.64:0/2101483619 >> v1:172.21.15.64:6790/0 conn(0x7f8a64135b50 legacy=0x7f8a6413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:50:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 1 monclient: found mon.noname-d 2024-09-17T15:50:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:50:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 1 -- 172.21.15.64:0/2101483619 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6409ccd0 con 0x7f8a64135f70 2024-09-17T15:50:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 1 -- 172.21.15.64:0/2101483619 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8a5c001bb0 con 0x7f8a64135f70 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a6abd3640 5 monclient: authenticate success, global_id 124260 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a6abd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient: dump: 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.052 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a6abd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 1 -- 172.21.15.64:0/2101483619 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8a5c001d50 con 0x7f8a64135f70 2024-09-17T15:50:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 1 -- 172.21.15.64:0/2101483619 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8a5c003cd0 con 0x7f8a64135f70 2024-09-17T15:50:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a6abd3640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a6abd3640 4 set_mon_vals no callback set 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.039+0000 7f8a5b7fe640 10 monclient: dump: 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.040+0000 7f8a6abd3640 10 monclient: shutdown 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.040+0000 7f8a6abd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.040+0000 7f8a6abd3640 1 -- 172.21.15.64:0/2101483619 >> v1:172.21.15.64:6792/0 conn(0x7f8a64135f70 legacy=0x7f8a641415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.040+0000 7f8a6abd3640 1 -- 172.21.15.64:0/2101483619 shutdown_connections 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.040+0000 7f8a6abd3640 1 -- 172.21.15.64:0/2101483619 >> 172.21.15.64:0/2101483619 conn(0x7f8a6406db30 msgr2=0x7f8a640ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.040+0000 7f8a6abd3640 1 -- 172.21.15.64:0/2101483619 shutdown_connections 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.040+0000 7f8a6abd3640 1 -- 172.21.15.64:0/2101483619 wait complete. 2024-09-17T15:50:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.040+0000 7f8a6abd3640 10 monclient: build_initial_monmap 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.040+0000 7f8a6abd3640 10 monclient: monmap: 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:18.056 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.041+0000 7f8a6abd3640 1 Processor -- start 2024-09-17T15:50:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.041+0000 7f8a6abd3640 1 -- start start 2024-09-17T15:50:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.041+0000 7f8a6abd3640 10 monclient: init 2024-09-17T15:50:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.041+0000 7f8a6abd3640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.041+0000 7f8a6abd3640 10 monclient: _add_conns ranks=[1,0,2,4,5,3,7,6,8] 2024-09-17T15:50:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.041+0000 7f8a6abd3640 10 monclient(hunting): picked mon.noname-f con 0x7f8a640a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.041+0000 7f8a6abd3640 10 monclient(hunting): picked mon.noname-a con 0x7f8a64135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:50:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.041+0000 7f8a6abd3640 10 monclient(hunting): picked mon.noname-b con 0x7f8a64135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.041+0000 7f8a6abd3640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a640a5310 con 0x7f8a64135b50 2024-09-17T15:50:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.041+0000 7f8a6abd3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a640a05b0 con 0x7f8a640a5520 2024-09-17T15:50:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.041+0000 7f8a6abd3640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a641324d0 con 0x7f8a64135f70 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.041+0000 7f8a6abd3640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.041+0000 7f8a6abd3640 10 monclient(hunting): authenticate will time out at 1015.716934s 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a69170640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f8a64135f70 0x7f8a641d5950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:34514/0 (socket says 172.21.15.64:34514) 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a69170640 1 -- 172.21.15.64:0/1477467941 learned_addr learned my addr 172.21.15.64:0/1477467941 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a5bfff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f8a64135b50 0x7f8a64131dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:48460/0 (socket says 172.21.15.64:48460) 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1320085753 0 0) 0x7f8a641324d0 con 0x7f8a64135f70 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): my global_id is 124301 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a50003520 con 0x7f8a64135f70 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2115821965 0 0) 0x7f8a640a5310 con 0x7f8a64135b50 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): my global_id is 124245 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a641324d0 con 0x7f8a64135b50 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3874687741 0 0) 0x7f8a50003520 con 0x7f8a64135f70 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8a640a5310 con 0x7f8a64135f70 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8a5c003970 con 0x7f8a64135f70 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): dump: 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1326243800 0 0) 0x7f8a640a05b0 con 0x7f8a640a5520 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): my global_id is 6031 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a50003520 con 0x7f8a640a5520 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1989307595 0 0) 0x7f8a641324d0 con 0x7f8a64135b50 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8a640a05b0 con 0x7f8a64135b50 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8a54001550 con 0x7f8a64135b50 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient(hunting): dump: 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.062 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2477711701 0 0) 0x7f8a640a5310 con 0x7f8a64135f70 2024-09-17T15:50:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 >> v1:172.21.15.100:6789/0 conn(0x7f8a640a5520 legacy=0x7f8a641316b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 >> v1:172.21.15.64:6789/0 conn(0x7f8a64135b50 legacy=0x7f8a64131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient: _finish_hunting 0 2024-09-17T15:50:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 monclient: found mon.b 2024-09-17T15:50:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a640a8c60 con 0x7f8a64135f70 2024-09-17T15:50:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient: _finish_auth 0 2024-09-17T15:50:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient: _check_auth_tickets 2024-09-17T15:50:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a6abd3640 5 monclient: authenticate success, global_id 124301 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a6abd3640 10 monclient: _renew_subs 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a6abd3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a6abd3640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8a64134ec0 con 0x7f8a64135f70 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a6abd3640 10 monclient: _renew_subs 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a6abd3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a6abd3640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8a640a02d0 con 0x7f8a64135f70 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8a5c001cf0 con 0x7f8a64135f70 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8a5c004180 con 0x7f8a64135f70 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a5affd640 4 set_mon_vals no callback set 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:18.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.042+0000 7f8a597fa640 10 monclient: dump: 2024-09-17T15:50:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.065 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.065 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.065 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.043+0000 7f8a6abd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:18.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.043+0000 7f8a6abd3640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8a440065c0 con 0x7f8a44002860 2024-09-17T15:50:18.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.043+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f8a5c004320 con 0x7f8a64135f70 2024-09-17T15:50:18.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(197..197 src has 1..197) ==== 3898+0+0 (unknown 1479433347 0 0) 0x7f8a5c04ef00 con 0x7f8a64135f70 2024-09-17T15:50:18.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1403722600 0 0) 0x7f8a440065c0 con 0x7f8a44002860 2024-09-17T15:50:18.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 10 monclient(hunting): my global_id is 124310 2024-09-17T15:50:18.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a640a5310 con 0x7f8a44002860 2024-09-17T15:50:18.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a5bfff640 10 monclient: get_auth_request con 0x7f8a50040b00 auth_method 0 2024-09-17T15:50:18.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1547753305 0 0) 0x7f8a640a5310 con 0x7f8a44002860 2024-09-17T15:50:18.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8a440065c0 con 0x7f8a44002860 2024-09-17T15:50:18.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8a60004200 con 0x7f8a44002860 2024-09-17T15:50:18.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:18.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 10 monclient: dump: 2024-09-17T15:50:18.066 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.066 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.066 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.066 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.066 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 87308729 0 0) 0x7f8a440065c0 con 0x7f8a44002860 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a4400a160 con 0x7f8a44002860 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8a60006540 con 0x7f8a44002860 2024-09-17T15:50:18.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:18.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:18.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 >> v1:172.21.15.64:6790/0 conn(0x7f8a44002860 legacy=0x7f8a44004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.044+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 >> v1:172.21.15.64:6790/0 conn(0x7f8a44002860 legacy=0x7f8a44004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:18.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.050+0000 7f8a6abd3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:18.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a6abd3640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8a44009e50 con 0x7f8a44009110 2024-09-17T15:50:18.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2928242933 0 0) 0x7f8a44009e50 con 0x7f8a44009110 2024-09-17T15:50:18.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 10 monclient(hunting): my global_id is 124319 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a440065c0 con 0x7f8a44009110 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3887009608 0 0) 0x7f8a440065c0 con 0x7f8a44009110 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8a44009e50 con 0x7f8a44009110 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8a60004c10 con 0x7f8a44009110 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 10 monclient: dump: 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.070 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1067772900 0 0) 0x7f8a44009e50 con 0x7f8a44009110 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 --> v1:172.21.15.64:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8a4400c3f0 con 0x7f8a44009110 2024-09-17T15:50:18.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7833 (unknown 0 0 3712459325) 0x7f8a600066f0 con 0x7f8a44009110 2024-09-17T15:50:18.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:18.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:18.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 >> v1:172.21.15.64:6790/0 conn(0x7f8a44009110 legacy=0x7f8a440095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.051+0000 7f8a597fa640 1 -- 172.21.15.64:0/1477467941 >> v1:172.21.15.64:6790/0 conn(0x7f8a44009110 legacy=0x7f8a440095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:18.072 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:18.072 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:18.072 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:18.072 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:50:18.072 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 28, 2024-09-17T15:50:18.072 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 24969, 2024-09-17T15:50:18.072 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:18.073 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:18.073 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:18.073 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:18.073 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:18.073 INFO:teuthology.orchestra.run.smithi064.stdout: 6 2024-09-17T15:50:18.073 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.073 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:50:18.073 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:18.073 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:18.073 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:18.073 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:18.073 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:18.073 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:18.074 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:18.075 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:18.076 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:18.077 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:18.079 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.082 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.082 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:18.082 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.082 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.082 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:18.082 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:18.082 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.082 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.082 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.083 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.084 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.084 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:18.084 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:18.084 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.084 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.084 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.084 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:18.084 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.084 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.085 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:18.085 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:18.085 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.085 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.085 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.085 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:18.085 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:18.085 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.085 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.086 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.087 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.088 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:18.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.091 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": [ 2024-09-17T15:50:18.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.091 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:18.091 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:18.091 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:18.091 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.091 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.091 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:18.091 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.052+0000 7f8a6abd3640 1 -- 172.21.15.64:0/1477467941 >> v1:172.21.15.100:6812/4210963484 conn(0x7f8a50040b00 legacy=0x7f8a50042fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.052+0000 7f8a6abd3640 10 monclient: shutdown 2024-09-17T15:50:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.052+0000 7f8a6abd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.052+0000 7f8a6abd3640 1 -- 172.21.15.64:0/1477467941 >> v1:172.21.15.64:6790/0 conn(0x7f8a64135f70 legacy=0x7f8a641d5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.052+0000 7f8a69170640 1 -- 172.21.15.64:0/1477467941 reap_dead start 2024-09-17T15:50:18.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.052+0000 7f8a69170640 1 -- 172.21.15.64:0/1477467941 reap_dead start 2024-09-17T15:50:18.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.052+0000 7f8a6abd3640 1 -- 172.21.15.64:0/1477467941 shutdown_connections 2024-09-17T15:50:18.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.052+0000 7f8a6abd3640 1 -- 172.21.15.64:0/1477467941 >> 172.21.15.64:0/1477467941 conn(0x7f8a6406db30 msgr2=0x7f8a640a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:18.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.052+0000 7f8a6abd3640 1 -- 172.21.15.64:0/1477467941 shutdown_connections 2024-09-17T15:50:18.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.052+0000 7f8a6abd3640 1 -- 172.21.15.64:0/1477467941 wait complete. 2024-09-17T15:50:18.093 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:50:18.093 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon d 2024-09-17T15:50:18.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-17T15:50:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.194+0000 7f43e0c40640 10 monclient: get_monmap_and_config 2024-09-17T15:50:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.194+0000 7f43e0c40640 10 monclient: build_initial_monmap 2024-09-17T15:50:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.194+0000 7f43e0c40640 10 monclient: monmap: 2024-09-17T15:50:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:18.195150+0000 2024-09-17T15:50:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:18.195150+0000 2024-09-17T15:50:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:18.197 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.194+0000 7f43e0c40640 1 Processor -- start 2024-09-17T15:50:18.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.194+0000 7f43e0c40640 1 -- start start 2024-09-17T15:50:18.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.194+0000 7f43e0c40640 10 monclient: init 2024-09-17T15:50:18.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43e0c40640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:18.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43e0c40640 10 monclient: _add_conns ranks=[5,4,2,1,6,0,8,3,7] 2024-09-17T15:50:18.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43e0c40640 10 monclient(hunting): picked mon.noname-h con 0x7f43dc0a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:50:18.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43e0c40640 10 monclient(hunting): picked mon.noname-c con 0x7f43dc135c80 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:18.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43e0c40640 10 monclient(hunting): picked mon.noname-b con 0x7f43dc1360a0 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:18.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43e0c40640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43dc0aacb0 con 0x7f43dc1360a0 2024-09-17T15:50:18.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43e0c40640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43dc09d780 con 0x7f43dc135c80 2024-09-17T15:50:18.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43e0c40640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43dc09da60 con 0x7f43dc0a54b0 2024-09-17T15:50:18.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43e0c40640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:18.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43e0c40640 10 monclient(hunting): authenticate will time out at 1015.870641s 2024-09-17T15:50:18.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43d9d9b640 1 -- >> v1:172.21.15.64:6791/0 conn(0x7f43dc135c80 legacy=0x7f43dc13df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:18.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43dad9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f43dc1360a0 0x7f43dc141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:34536/0 (socket says 172.21.15.64:34536) 2024-09-17T15:50:18.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43dad9d640 1 -- 172.21.15.64:0/2606145908 learned_addr learned my addr 172.21.15.64:0/2606145908 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:18.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43da59c640 1 -- 172.21.15.64:0/2606145908 >> v1:172.21.15.100:6791/0 conn(0x7f43dc0a54b0 legacy=0x7f43dc0a5890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:50:18.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.195+0000 7f43d959a640 1 -- 172.21.15.64:0/2606145908 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4112018207 0 0) 0x7f43dc0aacb0 con 0x7f43dc1360a0 2024-09-17T15:50:18.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient(hunting): my global_id is 124328 2024-09-17T15:50:18.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 1 -- 172.21.15.64:0/2606145908 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43c8003520 con 0x7f43dc1360a0 2024-09-17T15:50:18.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 1 -- 172.21.15.64:0/2606145908 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4009576006 0 0) 0x7f43c8003520 con 0x7f43dc1360a0 2024-09-17T15:50:18.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 1 -- 172.21.15.64:0/2606145908 >> v1:172.21.15.100:6791/0 conn(0x7f43dc0a54b0 legacy=0x7f43dc0a5890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 1 -- 172.21.15.64:0/2606145908 >> v1:172.21.15.64:6791/0 conn(0x7f43dc135c80 legacy=0x7f43dc13df10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient: _finish_hunting 0 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 1 monclient: found mon.noname-b 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 1 -- 172.21.15.64:0/2606145908 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43dc09dd40 con 0x7f43dc1360a0 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient: _finish_auth 0 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient: _check_auth_tickets 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43e0c40640 5 monclient: authenticate success, global_id 124328 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 1 -- 172.21.15.64:0/2606145908 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f43d4001830 con 0x7f43dc1360a0 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:50:18.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient: dump: 2024-09-17T15:50:18.208 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.208 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.208 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.208 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.208 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.208 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.208 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.208 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.208 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.208 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.208 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.208 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.209 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.209 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.209 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.209 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43e0c40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:18.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 1 -- 172.21.15.64:0/2606145908 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f43d40019d0 con 0x7f43dc1360a0 2024-09-17T15:50:18.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:18.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43e0c40640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:18.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 1 -- 172.21.15.64:0/2606145908 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f43d4004be0 con 0x7f43dc1360a0 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43e0c40640 4 set_mon_vals no callback set 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43d959a640 10 monclient: dump: 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.210 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43e0c40640 10 monclient: shutdown 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43e0c40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43e0c40640 1 -- 172.21.15.64:0/2606145908 >> v1:172.21.15.64:6790/0 conn(0x7f43dc1360a0 legacy=0x7f43dc141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43e0c40640 1 -- 172.21.15.64:0/2606145908 shutdown_connections 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43e0c40640 1 -- 172.21.15.64:0/2606145908 >> 172.21.15.64:0/2606145908 conn(0x7f43dc06daa0 msgr2=0x7f43dc0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.196+0000 7f43e0c40640 1 -- 172.21.15.64:0/2606145908 shutdown_connections 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.197+0000 7f43e0c40640 1 -- 172.21.15.64:0/2606145908 wait complete. 2024-09-17T15:50:18.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.197+0000 7f43e0c40640 10 monclient: build_initial_monmap 2024-09-17T15:50:18.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.197+0000 7f43e0c40640 10 monclient: monmap: 2024-09-17T15:50:18.212 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:18.212 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:18.212 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:18.212 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:18.212 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:18.212 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:18.212 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:18.212 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:18.212 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:18.212 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:18.212 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:18.212 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:18.213 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:18.213 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:18.213 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:18.213 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43e0c40640 1 Processor -- start 2024-09-17T15:50:18.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43e0c40640 1 -- start start 2024-09-17T15:50:18.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43e0c40640 10 monclient: init 2024-09-17T15:50:18.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43e0c40640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:18.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43e0c40640 10 monclient: _add_conns ranks=[5,3,7,1,4,8,2,0,6] 2024-09-17T15:50:18.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43e0c40640 10 monclient(hunting): picked mon.noname-h con 0x7f43dc0a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:50:18.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43e0c40640 10 monclient(hunting): picked mon.noname-g con 0x7f43dc135c80 addr v1:172.21.15.100:6790/0 2024-09-17T15:50:18.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43e0c40640 10 monclient(hunting): picked mon.noname-i con 0x7f43dc1360a0 addr v1:172.21.15.100:6792/0 2024-09-17T15:50:18.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43e0c40640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43dc0a50a0 con 0x7f43dc135c80 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43e0c40640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43dc0a31e0 con 0x7f43dc0a54b0 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43e0c40640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43dc0a34c0 con 0x7f43dc1360a0 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43e0c40640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43e0c40640 10 monclient(hunting): authenticate will time out at 1015.873639s 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43da59c640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7f43dc0a54b0 legacy=0x7f43dc09fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43dad9d640 1 -- >> v1:172.21.15.100:6792/0 conn(0x7f43dc1360a0 legacy=0x7f43dc1d5e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43d9d9b640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f43dc135c80 0x7f43dc0a0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:43522/0 (socket says 172.21.15.64:43522) 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.198+0000 7f43d9d9b640 1 -- 172.21.15.64:0/2661694313 learned_addr learned my addr 172.21.15.64:0/2661694313 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4007599014 0 0) 0x7f43c4003e10 con 0x7f43dc135c80 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient(hunting): my global_id is 114960 2024-09-17T15:50:18.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43bc003520 con 0x7f43dc135c80 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 163589350 0 0) 0x7f43bc003520 con 0x7f43dc135c80 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43c4003e10 con 0x7f43dc135c80 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f43c40048c0 con 0x7f43dc135c80 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient(hunting): dump: 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.215 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2458739809 0 0) 0x7f43c4003e10 con 0x7f43dc135c80 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 >> v1:172.21.15.100:6792/0 conn(0x7f43dc1360a0 legacy=0x7f43dc1d5e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 >> v1:172.21.15.100:6791/0 conn(0x7f43dc0a54b0 legacy=0x7f43dc09fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 1 monclient: found mon.g 2024-09-17T15:50:18.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:18.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:50:18.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43dc135060 con 0x7f43dc135c80 2024-09-17T15:50:18.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:50:18.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:50:18.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:18.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43e0c40640 5 monclient: authenticate success, global_id 114960 2024-09-17T15:50:18.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43e0c40640 10 monclient: _renew_subs 2024-09-17T15:50:18.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43e0c40640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:50:18.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43e0c40640 1 -- 172.21.15.64:0/2661694313 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f43dc0a2930 con 0x7f43dc135c80 2024-09-17T15:50:18.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43e0c40640 10 monclient: _renew_subs 2024-09-17T15:50:18.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43e0c40640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:50:18.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43e0c40640 1 -- 172.21.15.64:0/2661694313 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f43dc0a0840 con 0x7f43dc135c80 2024-09-17T15:50:18.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.3 v1:172.21.15.100:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f43c4004eb0 con 0x7f43dc135c80 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.3 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f43c40053e0 con 0x7f43dc135c80 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d8d99640 4 set_mon_vals no callback set 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.199+0000 7f43d2ffd640 10 monclient: dump: 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.218 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.219 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.219 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.219 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.219 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.219 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.219 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.219 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.219 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.200+0000 7f43e0c40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:18.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.200+0000 7f43e0c40640 1 -- 172.21.15.64:0/2661694313 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f43c00065c0 con 0x7f43c0002860 2024-09-17T15:50:18.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.200+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.3 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f43c4011cc0 con 0x7f43dc135c80 2024-09-17T15:50:18.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3790147309 0 0) 0x7f43c00065c0 con 0x7f43c0002860 2024-09-17T15:50:18.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 10 monclient(hunting): my global_id is 124269 2024-09-17T15:50:18.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43c4003e10 con 0x7f43c0002860 2024-09-17T15:50:18.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.3 v1:172.21.15.100:6790/0 8 ==== osd_map(197..197 src has 1..197) ==== 3898+0+0 (unknown 1479433347 0 0) 0x7f43c404f220 con 0x7f43dc135c80 2024-09-17T15:50:18.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43dad9d640 10 monclient: get_auth_request con 0x7f43bc0401a0 auth_method 0 2024-09-17T15:50:18.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1550943957 0 0) 0x7f43c4003e10 con 0x7f43c0002860 2024-09-17T15:50:18.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43c00065c0 con 0x7f43c0002860 2024-09-17T15:50:18.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f43cc004110 con 0x7f43c0002860 2024-09-17T15:50:18.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:50:18.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 10 monclient: dump: 2024-09-17T15:50:18.220 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.221 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.222 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1005748565 0 0) 0x7f43c00065c0 con 0x7f43c0002860 2024-09-17T15:50:18.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 --> v1:172.21.15.64:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43c000a160 con 0x7f43c0002860 2024-09-17T15:50:18.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f43cc006530 con 0x7f43c0002860 2024-09-17T15:50:18.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:18.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:18.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 >> v1:172.21.15.64:6792/0 conn(0x7f43c0002860 legacy=0x7f43c0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.201+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 >> v1:172.21.15.64:6792/0 conn(0x7f43c0002860 legacy=0x7f43c0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:18.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.207+0000 7f43e0c40640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:18.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.207+0000 7f43e0c40640 1 -- 172.21.15.64:0/2661694313 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f43c0009e50 con 0x7f43c0009110 2024-09-17T15:50:18.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1901273127 0 0) 0x7f43c0009e50 con 0x7f43c0009110 2024-09-17T15:50:18.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 10 monclient(hunting): my global_id is 124278 2024-09-17T15:50:18.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43c00065c0 con 0x7f43c0009110 2024-09-17T15:50:18.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2868206299 0 0) 0x7f43c00065c0 con 0x7f43c0009110 2024-09-17T15:50:18.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43c0009e50 con 0x7f43c0009110 2024-09-17T15:50:18.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f43cc004b90 con 0x7f43c0009110 2024-09-17T15:50:18.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:50:18.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 10 monclient: dump: 2024-09-17T15:50:18.225 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.225 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.225 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.225 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.225 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.225 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.225 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.225 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.225 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 819564980 0 0) 0x7f43c0009e50 con 0x7f43c0009110 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 --> v1:172.21.15.64:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f43c000c3f0 con 0x7f43c0009110 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7499 (unknown 0 0 4184765711) 0x7f43cc006120 con 0x7f43c0009110 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 >> v1:172.21.15.64:6792/0 conn(0x7f43c0009110 legacy=0x7f43c00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.208+0000 7f43d2ffd640 1 -- 172.21.15.64:0/2661694313 >> v1:172.21.15.64:6792/0 conn(0x7f43c0009110 legacy=0x7f43c00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:18.227 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:18.227 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:18.227 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:18.227 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:50:18.227 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 28, 2024-09-17T15:50:18.227 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 25126, 2024-09-17T15:50:18.227 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:18.227 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:18.227 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:18.227 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: 6 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:18.228 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:18.230 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:18.231 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:18.231 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:18.231 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:18.231 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:18.231 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:18.231 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:18.231 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:18.231 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:18.231 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:18.231 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:18.232 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:18.232 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:18.232 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:18.232 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:18.232 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:18.232 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:18.232 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:18.232 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:18.232 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:18.232 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:18.232 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:18.232 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.233 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:18.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.233 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:18.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.233 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:18.233 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:18.233 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.233 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.233 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.233 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:18.233 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:18.235 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:18.237 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.239 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.240 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:18.240 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.240 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.240 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.240 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:18.240 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:18.240 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.240 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.240 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.240 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:18.241 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:18.241 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.241 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:18.241 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.241 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:18.241 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:18.241 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.242 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.243 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:18.243 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:18.243 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.243 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.243 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.243 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:18.243 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:18.243 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.243 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.244 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:18.245 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:18.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.245 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:18.245 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:18.245 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:18.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.245 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.245 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:18.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.245 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:18.245 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:18.245 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:18.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.246 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.246 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:18.246 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:18.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.209+0000 7f43e0c40640 1 -- 172.21.15.64:0/2661694313 >> v1:172.21.15.100:6812/4210963484 conn(0x7f43bc0401a0 legacy=0x7f43bc042660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.209+0000 7f43e0c40640 10 monclient: shutdown 2024-09-17T15:50:18.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.209+0000 7f43e0c40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:18.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.209+0000 7f43e0c40640 1 -- 172.21.15.64:0/2661694313 >> v1:172.21.15.100:6790/0 conn(0x7f43dc135c80 legacy=0x7f43dc0a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.209+0000 7f43dad9d640 1 -- 172.21.15.64:0/2661694313 reap_dead start 2024-09-17T15:50:18.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.209+0000 7f43dad9d640 1 -- 172.21.15.64:0/2661694313 reap_dead start 2024-09-17T15:50:18.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.209+0000 7f43e0c40640 1 -- 172.21.15.64:0/2661694313 shutdown_connections 2024-09-17T15:50:18.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.209+0000 7f43e0c40640 1 -- 172.21.15.64:0/2661694313 >> 172.21.15.64:0/2661694313 conn(0x7f43dc06daa0 msgr2=0x7f43dc0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:18.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.209+0000 7f43e0c40640 1 -- 172.21.15.64:0/2661694313 shutdown_connections 2024-09-17T15:50:18.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.209+0000 7f43e0c40640 1 -- 172.21.15.64:0/2661694313 wait complete. 2024-09-17T15:50:18.248 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:50:18.248 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon f 2024-09-17T15:50:18.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-17T15:50:18.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.348+0000 7f43f8bbb640 10 monclient: get_monmap_and_config 2024-09-17T15:50:18.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.348+0000 7f43f8bbb640 10 monclient: build_initial_monmap 2024-09-17T15:50:18.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.348+0000 7f43f8bbb640 10 monclient: monmap: 2024-09-17T15:50:18.350 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:18.350 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:18.349649+0000 2024-09-17T15:50:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:18.349649+0000 2024-09-17T15:50:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:18.352 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.349+0000 7f43f8bbb640 1 Processor -- start 2024-09-17T15:50:18.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.349+0000 7f43f8bbb640 1 -- start start 2024-09-17T15:50:18.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.349+0000 7f43f8bbb640 10 monclient: init 2024-09-17T15:50:18.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f8bbb640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f8bbb640 10 monclient: _add_conns ranks=[1,7,8,0,2,6,5,3,4] 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f8bbb640 10 monclient(hunting): picked mon.noname-f con 0x7f43f40a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f8bbb640 10 monclient(hunting): picked mon.noname-i con 0x7f43f4135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f8bbb640 10 monclient(hunting): picked mon.noname-e con 0x7f43f4135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f8bbb640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f409d6f0 con 0x7f43f40a5520 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f8bbb640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f409d9d0 con 0x7f43f4135b50 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f8bbb640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f409dcb0 con 0x7f43f4135f70 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f8bbb640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f8bbb640 10 monclient(hunting): authenticate will time out at 1016.025144s 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f2d9d640 1 -- >> v1:172.21.15.64:6793/0 conn(0x7f43f4135f70 legacy=0x7f43f41415e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f1d9b640 1 -- >> v1:172.21.15.100:6792/0 conn(0x7f43f4135b50 legacy=0x7f43f413deb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f259c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f43f40a5520 0x7f43f40a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.64:51782/0 (socket says 172.21.15.64:51782) 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f259c640 1 -- 172.21.15.64:0/2456922549 learned_addr learned my addr 172.21.15.64:0/2456922549 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:18.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 1 -- 172.21.15.64:0/2456922549 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 230218291 0 0) 0x7f43f409d6f0 con 0x7f43f40a5520 2024-09-17T15:50:18.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 10 monclient(hunting): my global_id is 6040 2024-09-17T15:50:18.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 1 -- 172.21.15.64:0/2456922549 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43e0003520 con 0x7f43f40a5520 2024-09-17T15:50:18.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 1 -- 172.21.15.64:0/2456922549 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2324307270 0 0) 0x7f43e0003520 con 0x7f43f40a5520 2024-09-17T15:50:18.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 1 -- 172.21.15.64:0/2456922549 >> v1:172.21.15.64:6793/0 conn(0x7f43f4135f70 legacy=0x7f43f41415e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:18.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 1 -- 172.21.15.64:0/2456922549 >> v1:172.21.15.100:6792/0 conn(0x7f43f4135b50 legacy=0x7f43f413deb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:18.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 10 monclient: _finish_hunting 0 2024-09-17T15:50:18.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 1 monclient: found mon.noname-f 2024-09-17T15:50:18.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:18.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.100:6789/0 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 1 -- 172.21.15.64:0/2456922549 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f409ccd0 con 0x7f43f40a5520 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 10 monclient: _finish_auth 0 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 10 monclient: _check_auth_tickets 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 1 -- 172.21.15.64:0/2456922549 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f43ec001bb0 con 0x7f43f40a5520 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f8bbb640 5 monclient: authenticate success, global_id 6040 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f8bbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.350+0000 7f43f159a640 10 monclient: dump: 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.361 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f8bbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f159a640 1 -- 172.21.15.64:0/2456922549 <== mon.1 v1:172.21.15.100:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f43ec001dd0 con 0x7f43f40a5520 2024-09-17T15:50:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f8bbb640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f8bbb640 4 set_mon_vals no callback set 2024-09-17T15:50:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f159a640 1 -- 172.21.15.64:0/2456922549 <== mon.1 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f43ec0020c0 con 0x7f43f40a5520 2024-09-17T15:50:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f159a640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f159a640 10 monclient: dump: 2024-09-17T15:50:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.363 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f8bbb640 10 monclient: shutdown 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f8bbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2456922549 >> v1:172.21.15.100:6789/0 conn(0x7f43f40a5520 legacy=0x7f43f40a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2456922549 shutdown_connections 2024-09-17T15:50:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2456922549 >> 172.21.15.64:0/2456922549 conn(0x7f43f406db30 msgr2=0x7f43f40ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2456922549 shutdown_connections 2024-09-17T15:50:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.351+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2456922549 wait complete. 2024-09-17T15:50:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.352+0000 7f43f8bbb640 10 monclient: build_initial_monmap 2024-09-17T15:50:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.352+0000 7f43f8bbb640 10 monclient: monmap: 2024-09-17T15:50:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:18.366 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:18.366 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:18.366 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:18.366 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:18.366 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:18.366 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:18.366 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:18.366 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:18.366 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:18.366 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:18.366 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.366 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.352+0000 7f43f8bbb640 1 Processor -- start 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.352+0000 7f43f8bbb640 1 -- start start 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.352+0000 7f43f8bbb640 10 monclient: init 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 10 monclient: _add_conns ranks=[4,6,8,2,5,0,3,1,7] 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 10 monclient(hunting): picked mon.noname-c con 0x7f43f40a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 10 monclient(hunting): picked mon.noname-d con 0x7f43f4135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 10 monclient(hunting): picked mon.noname-e con 0x7f43f4135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f40a35d0 con 0x7f43f40a5520 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f4134d80 con 0x7f43f4135b50 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f259c640 1 -- >> v1:172.21.15.64:6791/0 conn(0x7f43f40a5520 legacy=0x7f43f40a4b00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6791/0 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f40a1270 con 0x7f43f4135f70 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 10 monclient(hunting): authenticate will time out at 1016.028193s 2024-09-17T15:50:18.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f2d9d640 1 -- >> v1:172.21.15.64:6793/0 conn(0x7f43f4135f70 legacy=0x7f43f41d60e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:50:18.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f1d9b640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f43f4135b50 0x7f43f41d29b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:52016/0 (socket says 172.21.15.64:52016) 2024-09-17T15:50:18.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f1d9b640 1 -- 172.21.15.64:0/2093552625 learned_addr learned my addr 172.21.15.64:0/2093552625 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:18.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2578625625 0 0) 0x7f43f4134d80 con 0x7f43f4135b50 2024-09-17T15:50:18.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 10 monclient(hunting): my global_id is 124287 2024-09-17T15:50:18.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43d4003520 con 0x7f43f4135b50 2024-09-17T15:50:18.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 750011373 0 0) 0x7f43d4003520 con 0x7f43f4135b50 2024-09-17T15:50:18.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43f4134d80 con 0x7f43f4135b50 2024-09-17T15:50:18.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f43dc0043a0 con 0x7f43f4135b50 2024-09-17T15:50:18.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:18.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:50:18.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 10 monclient(hunting): dump: 2024-09-17T15:50:18.369 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.369 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.369 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.369 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.369 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.369 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.369 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.369 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.369 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.369 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.369 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.369 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.370 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.370 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.370 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.370 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:18.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1513109864 0 0) 0x7f43f4134d80 con 0x7f43f4135b50 2024-09-17T15:50:18.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 >> v1:172.21.15.64:6793/0 conn(0x7f43f4135f70 legacy=0x7f43f41d60e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:18.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 >> v1:172.21.15.64:6791/0 conn(0x7f43f40a5520 legacy=0x7f43f40a4b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:18.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 10 monclient: _finish_hunting 0 2024-09-17T15:50:18.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 1 monclient: found mon.d 2024-09-17T15:50:18.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:18.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:50:18.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f40a32f0 con 0x7f43f4135b50 2024-09-17T15:50:18.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 10 monclient: _finish_auth 0 2024-09-17T15:50:18.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:50:18.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:18.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 5 monclient: authenticate success, global_id 124287 2024-09-17T15:50:18.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 10 monclient: _renew_subs 2024-09-17T15:50:18.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:50:18.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2093552625 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f43f40a1f60 con 0x7f43f4135b50 2024-09-17T15:50:18.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 10 monclient: _renew_subs 2024-09-17T15:50:18.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:50:18.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.353+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2093552625 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f43f41ca130 con 0x7f43f4135b50 2024-09-17T15:50:18.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.354+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f43dc0045c0 con 0x7f43f4135b50 2024-09-17T15:50:18.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.354+0000 7f43eaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:18.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.354+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f43dc004b70 con 0x7f43f4135b50 2024-09-17T15:50:18.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.354+0000 7f43eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.354+0000 7f43eaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:50:18.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.354+0000 7f43f0d99640 4 set_mon_vals no callback set 2024-09-17T15:50:18.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.354+0000 7f43eaffd640 10 monclient: dump: 2024-09-17T15:50:18.372 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.372 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.372 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.372 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.372 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.372 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.372 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.372 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.373 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.354+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f43dc004d90 con 0x7f43f4135b50 2024-09-17T15:50:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.354+0000 7f43f8bbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.354+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2093552625 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f43d80065c0 con 0x7f43d8002860 2024-09-17T15:50:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(197..197 src has 1..197) ==== 3898+0+0 (unknown 1479433347 0 0) 0x7f43dc04eda0 con 0x7f43f4135b50 2024-09-17T15:50:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43f2d9d640 10 monclient: get_auth_request con 0x7f43d4040140 auth_method 0 2024-09-17T15:50:18.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 428297477 0 0) 0x7f43d80065c0 con 0x7f43d8002860 2024-09-17T15:50:18.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 10 monclient(hunting): my global_id is 6049 2024-09-17T15:50:18.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43f4134d80 con 0x7f43d8002860 2024-09-17T15:50:18.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2707590247 0 0) 0x7f43f4134d80 con 0x7f43d8002860 2024-09-17T15:50:18.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43d80065c0 con 0x7f43d8002860 2024-09-17T15:50:18.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f43ec0020c0 con 0x7f43d8002860 2024-09-17T15:50:18.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:18.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 10 monclient: dump: 2024-09-17T15:50:18.374 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.375 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.375 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.375 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.375 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.375 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.375 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.375 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.375 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.375 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.375 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.375 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.375 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.375 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.376 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.376 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2482496798 0 0) 0x7f43d80065c0 con 0x7f43d8002860 2024-09-17T15:50:18.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.355+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 --> v1:172.21.15.100:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43d800a160 con 0x7f43d8002860 2024-09-17T15:50:18.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.356+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f43ec001b70 con 0x7f43d8002860 2024-09-17T15:50:18.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.356+0000 7f43eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:18.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.356+0000 7f43eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:18.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.356+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 >> v1:172.21.15.100:6789/0 conn(0x7f43d8002860 legacy=0x7f43d8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.356+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 >> v1:172.21.15.100:6789/0 conn(0x7f43d8002860 legacy=0x7f43d8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:18.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.362+0000 7f43f8bbb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:18.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.362+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2093552625 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f43d8009e50 con 0x7f43d8009110 2024-09-17T15:50:18.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1291258855 0 0) 0x7f43d8009e50 con 0x7f43d8009110 2024-09-17T15:50:18.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 10 monclient(hunting): my global_id is 6058 2024-09-17T15:50:18.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43d80065c0 con 0x7f43d8009110 2024-09-17T15:50:18.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4070381607 0 0) 0x7f43d80065c0 con 0x7f43d8009110 2024-09-17T15:50:18.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43d8009e50 con 0x7f43d8009110 2024-09-17T15:50:18.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f43ec0053c0 con 0x7f43d8009110 2024-09-17T15:50:18.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:18.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 10 monclient: dump: 2024-09-17T15:50:18.379 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.379 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.379 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.379 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.379 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.379 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.379 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.379 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.379 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.379 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.380 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.380 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.380 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.380 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.380 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.380 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3950613972 0 0) 0x7f43d8009e50 con 0x7f43d8009110 2024-09-17T15:50:18.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.363+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 --> v1:172.21.15.100:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f43d800c3f0 con 0x7f43d8009110 2024-09-17T15:50:18.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7669 (unknown 0 0 1154256042) 0x7f43ec007e00 con 0x7f43d8009110 2024-09-17T15:50:18.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:18.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:18.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 >> v1:172.21.15.100:6789/0 conn(0x7f43d8009110 legacy=0x7f43d80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43eaffd640 1 -- 172.21.15.64:0/2093552625 >> v1:172.21.15.100:6789/0 conn(0x7f43d8009110 legacy=0x7f43d80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:18.381 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "leader", 2024-09-17T15:50:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 28, 2024-09-17T15:50:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 275015, 2024-09-17T15:50:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:18.381 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: 6 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:18.382 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:18.383 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:18.384 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:18.385 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:18.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:18.387 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:18.388 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:18.389 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.390 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.392 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:18.393 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:18.394 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.397 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.397 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:18.397 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.397 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.397 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:18.397 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:18.397 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.397 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.397 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.398 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:18.399 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:18.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.400 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:18.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.400 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:18.400 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:18.400 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 4 2024-09-17T15:50:18.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.400 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.400 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:18.400 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:18.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2093552625 >> v1:172.21.15.100:6812/4210963484 conn(0x7f43d4040140 legacy=0x7f43d4042600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43f8bbb640 10 monclient: shutdown 2024-09-17T15:50:18.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43f8bbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:18.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2093552625 >> v1:172.21.15.64:6792/0 conn(0x7f43f4135b50 legacy=0x7f43f41d29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43f2d9d640 1 -- 172.21.15.64:0/2093552625 reap_dead start 2024-09-17T15:50:18.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43f2d9d640 1 -- 172.21.15.64:0/2093552625 reap_dead start 2024-09-17T15:50:18.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2093552625 shutdown_connections 2024-09-17T15:50:18.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2093552625 >> 172.21.15.64:0/2093552625 conn(0x7f43f406db30 msgr2=0x7f43f41334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:18.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2093552625 shutdown_connections 2024-09-17T15:50:18.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.364+0000 7f43f8bbb640 1 -- 172.21.15.64:0/2093552625 wait complete. 2024-09-17T15:50:18.403 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:50:18.403 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon g 2024-09-17T15:50:18.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-17T15:50:18.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.512+0000 7fcf6dcfe640 10 monclient: get_monmap_and_config 2024-09-17T15:50:18.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.512+0000 7fcf6dcfe640 10 monclient: build_initial_monmap 2024-09-17T15:50:18.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.512+0000 7fcf6dcfe640 10 monclient: monmap: 2024-09-17T15:50:18.514 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:18.514 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.514 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:50:18.513567+0000 2024-09-17T15:50:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:50:18.513567+0000 2024-09-17T15:50:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:18.515 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.513+0000 7fcf6dcfe640 1 Processor -- start 2024-09-17T15:50:18.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.513+0000 7fcf6dcfe640 1 -- start start 2024-09-17T15:50:18.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.513+0000 7fcf6dcfe640 10 monclient: init 2024-09-17T15:50:18.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.513+0000 7fcf6dcfe640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:18.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.513+0000 7fcf6dcfe640 10 monclient: _add_conns ranks=[0,2,6,7,4,3,8,1,5] 2024-09-17T15:50:18.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.513+0000 7fcf6dcfe640 10 monclient(hunting): picked mon.noname-a con 0x7fcf680a3f50 addr v1:172.21.15.64:6789/0 2024-09-17T15:50:18.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf6dcfe640 10 monclient(hunting): picked mon.noname-b con 0x7fcf680a01e0 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:18.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf6dcfe640 10 monclient(hunting): picked mon.noname-d con 0x7fcf680a0670 addr v1:172.21.15.64:6792/0 2024-09-17T15:50:18.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf6dcfe640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf680a4a20 con 0x7fcf680a3f50 2024-09-17T15:50:18.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf6dcfe640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf6809d720 con 0x7fcf680a01e0 2024-09-17T15:50:18.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf6dcfe640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf680a4c20 con 0x7fcf680a0670 2024-09-17T15:50:18.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf6dcfe640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:18.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf6dcfe640 10 monclient(hunting): authenticate will time out at 1016.189091s 2024-09-17T15:50:18.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf66ffd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fcf680a01e0 0x7fcf6813b650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:34548/0 (socket says 172.21.15.64:34548) 2024-09-17T15:50:18.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf677fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fcf680a3f50 0x7fcf6809fad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:48470/0 (socket says 172.21.15.64:48470) 2024-09-17T15:50:18.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf66ffd640 1 -- 172.21.15.64:0/3616915049 learned_addr learned my addr 172.21.15.64:0/3616915049 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:18.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 -- 172.21.15.64:0/3616915049 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282259755 0 0) 0x7fcf6809d720 con 0x7fcf680a01e0 2024-09-17T15:50:18.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient(hunting): my global_id is 124337 2024-09-17T15:50:18.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 -- 172.21.15.64:0/3616915049 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf50003520 con 0x7fcf680a01e0 2024-09-17T15:50:18.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 -- 172.21.15.64:0/3616915049 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2109855659 0 0) 0x7fcf680a4c20 con 0x7fcf680a0670 2024-09-17T15:50:18.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient(hunting): my global_id is 124296 2024-09-17T15:50:18.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 -- 172.21.15.64:0/3616915049 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf6809d720 con 0x7fcf680a0670 2024-09-17T15:50:18.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 -- 172.21.15.64:0/3616915049 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3495395497 0 0) 0x7fcf680a4a20 con 0x7fcf680a3f50 2024-09-17T15:50:18.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient(hunting): my global_id is 124254 2024-09-17T15:50:18.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 -- 172.21.15.64:0/3616915049 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf680a4c20 con 0x7fcf680a3f50 2024-09-17T15:50:18.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 -- 172.21.15.64:0/3616915049 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 417642282 0 0) 0x7fcf50003520 con 0x7fcf680a01e0 2024-09-17T15:50:18.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 -- 172.21.15.64:0/3616915049 >> v1:172.21.15.64:6792/0 conn(0x7fcf680a0670 legacy=0x7fcf6813ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 -- 172.21.15.64:0/3616915049 >> v1:172.21.15.64:6789/0 conn(0x7fcf680a3f50 legacy=0x7fcf6809fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient: _finish_hunting 0 2024-09-17T15:50:18.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 monclient: found mon.noname-b 2024-09-17T15:50:18.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:18.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:50:18.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 -- 172.21.15.64:0/3616915049 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf6809b560 con 0x7fcf680a01e0 2024-09-17T15:50:18.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient: _finish_auth 0 2024-09-17T15:50:18.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient: _check_auth_tickets 2024-09-17T15:50:18.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:18.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf6dcfe640 5 monclient: authenticate success, global_id 124337 2024-09-17T15:50:18.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 -- 172.21.15.64:0/3616915049 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcf5c001830 con 0x7fcf680a01e0 2024-09-17T15:50:18.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:50:18.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient: dump: 2024-09-17T15:50:18.527 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.527 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.527 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.527 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf6dcfe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:18.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 -- 172.21.15.64:0/3616915049 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf5c0019d0 con 0x7fcf680a01e0 2024-09-17T15:50:18.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:18.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 1 -- 172.21.15.64:0/3616915049 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcf5c004be0 con 0x7fcf680a01e0 2024-09-17T15:50:18.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf6dcfe640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:18.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf6dcfe640 4 set_mon_vals no callback set 2024-09-17T15:50:18.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:18.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.514+0000 7fcf667fc640 10 monclient: dump: 2024-09-17T15:50:18.529 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.529 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.529 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.529 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.529 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.529 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.515+0000 7fcf6dcfe640 10 monclient: shutdown 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.515+0000 7fcf6dcfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.515+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/3616915049 >> v1:172.21.15.64:6790/0 conn(0x7fcf680a01e0 legacy=0x7fcf6813b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.515+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/3616915049 shutdown_connections 2024-09-17T15:50:18.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.515+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/3616915049 >> 172.21.15.64:0/3616915049 conn(0x7fcf680a53c0 msgr2=0x7fcf680a57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.515+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/3616915049 shutdown_connections 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.515+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/3616915049 wait complete. 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.516+0000 7fcf6dcfe640 10 monclient: build_initial_monmap 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.516+0000 7fcf6dcfe640 10 monclient: monmap: 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:50:18.531 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:50:18.532 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:50:18.532 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:50:18.532 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:50:18.532 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:50:18.532 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:50:18.532 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.516+0000 7fcf6dcfe640 1 Processor -- start 2024-09-17T15:50:18.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.516+0000 7fcf6dcfe640 1 -- start start 2024-09-17T15:50:18.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.516+0000 7fcf6dcfe640 10 monclient: init 2024-09-17T15:50:18.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf6dcfe640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:18.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf6dcfe640 10 monclient: _add_conns ranks=[1,8,2,4,3,7,5,0,6] 2024-09-17T15:50:18.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf6dcfe640 10 monclient(hunting): picked mon.noname-f con 0x7fcf680a01e0 addr v1:172.21.15.100:6789/0 2024-09-17T15:50:18.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf6dcfe640 10 monclient(hunting): picked mon.noname-e con 0x7fcf680a0670 addr v1:172.21.15.64:6793/0 2024-09-17T15:50:18.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf6dcfe640 10 monclient(hunting): picked mon.noname-b con 0x7fcf680a3f50 addr v1:172.21.15.64:6790/0 2024-09-17T15:50:18.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf6dcfe640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf6808fe00 con 0x7fcf680a01e0 2024-09-17T15:50:18.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf66ffd640 1 -- >> v1:172.21.15.64:6793/0 conn(0x7fcf680a0670 legacy=0x7fcf6808f6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:50:18.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf6dcfe640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf6809f030 con 0x7fcf680a3f50 2024-09-17T15:50:18.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf6dcfe640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf68092120 con 0x7fcf680a0670 2024-09-17T15:50:18.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf6dcfe640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:18.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf6dcfe640 10 monclient(hunting): authenticate will time out at 1016.192348s 2024-09-17T15:50:18.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf677fe640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fcf680a01e0 0x7fcf6808efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.64:51810/0 (socket says 172.21.15.64:51810) 2024-09-17T15:50:18.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf67fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fcf680a3f50 0x7fcf681d1790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:34558/0 (socket says 172.21.15.64:34558) 2024-09-17T15:50:18.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf677fe640 1 -- 172.21.15.64:0/1891632472 learned_addr learned my addr 172.21.15.64:0/1891632472 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:50:18.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1961080002 0 0) 0x7fcf6809f030 con 0x7fcf680a3f50 2024-09-17T15:50:18.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 10 monclient(hunting): my global_id is 124346 2024-09-17T15:50:18.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf54003520 con 0x7fcf680a3f50 2024-09-17T15:50:18.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3790478611 0 0) 0x7fcf6808fe00 con 0x7fcf680a01e0 2024-09-17T15:50:18.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 10 monclient(hunting): my global_id is 6067 2024-09-17T15:50:18.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf6809f030 con 0x7fcf680a01e0 2024-09-17T15:50:18.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 341585483 0 0) 0x7fcf54003520 con 0x7fcf680a3f50 2024-09-17T15:50:18.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf6808fe00 con 0x7fcf680a3f50 2024-09-17T15:50:18.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcf60001990 con 0x7fcf680a3f50 2024-09-17T15:50:18.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 10 monclient(hunting): dump: 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.535 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.517+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2001885616 0 0) 0x7fcf6809f030 con 0x7fcf680a01e0 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf54003520 con 0x7fcf680a01e0 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcf58001990 con 0x7fcf680a01e0 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 10 monclient(hunting): dump: 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.536 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3148236392 0 0) 0x7fcf6808fe00 con 0x7fcf680a3f50 2024-09-17T15:50:18.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 >> v1:172.21.15.64:6793/0 conn(0x7fcf680a0670 legacy=0x7fcf6808f6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:18.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 >> v1:172.21.15.100:6789/0 conn(0x7fcf680a01e0 legacy=0x7fcf6808efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:50:18.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 1 monclient: found mon.b 2024-09-17T15:50:18.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:18.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:18.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf68091e40 con 0x7fcf680a3f50 2024-09-17T15:50:18.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 10 monclient: _finish_auth 0 2024-09-17T15:50:18.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:50:18.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:18.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf6dcfe640 5 monclient: authenticate success, global_id 124346 2024-09-17T15:50:18.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf6dcfe640 10 monclient: _renew_subs 2024-09-17T15:50:18.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf6dcfe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:18.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcf680905e0 con 0x7fcf680a3f50 2024-09-17T15:50:18.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf6dcfe640 10 monclient: _renew_subs 2024-09-17T15:50:18.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf6dcfe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:50:18.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcf68092be0 con 0x7fcf680a3f50 2024-09-17T15:50:18.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf60003c20 con 0x7fcf680a3f50 2024-09-17T15:50:18.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:18.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcf60004150 con 0x7fcf680a3f50 2024-09-17T15:50:18.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf65ffb640 4 set_mon_vals no callback set 2024-09-17T15:50:18.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:18.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.518+0000 7fcf4ffff640 10 monclient: dump: 2024-09-17T15:50:18.539 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.539 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.540 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.540 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.540 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.540 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.540 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.540 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.540 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.540 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.540 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.540 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.540 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.540 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.540 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.519+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fcf600042f0 con 0x7fcf680a3f50 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.519+0000 7fcf6dcfe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.519+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcf480065c0 con 0x7fcf48002860 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.519+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(198..198 src has 1..198) ==== 3898+0+0 (unknown 1724120112 0 0) 0x7fcf6004efb0 con 0x7fcf680a3f50 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.519+0000 7fcf677fe640 10 monclient: get_auth_request con 0x7fcf540408d0 auth_method 0 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.519+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3349533296 0 0) 0x7fcf480065c0 con 0x7fcf48002860 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.519+0000 7fcf4ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.519+0000 7fcf4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.519+0000 7fcf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.519+0000 7fcf4ffff640 10 monclient(hunting): my global_id is 114969 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.519+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf6808fe00 con 0x7fcf48002860 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.520+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 826140594 0 0) 0x7fcf6808fe00 con 0x7fcf48002860 2024-09-17T15:50:18.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.520+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf480065c0 con 0x7fcf48002860 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.520+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcf5c004b20 con 0x7fcf48002860 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.520+0000 7fcf4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.520+0000 7fcf4ffff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.520+0000 7fcf4ffff640 10 monclient: dump: 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.542 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.543 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.543 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.543 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.543 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.543 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.543 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.520+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1765324732 0 0) 0x7fcf480065c0 con 0x7fcf48002860 2024-09-17T15:50:18.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.520+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.100:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf4800a160 con 0x7fcf48002860 2024-09-17T15:50:18.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.520+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcf5c001210 con 0x7fcf48002860 2024-09-17T15:50:18.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.520+0000 7fcf4ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:18.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.520+0000 7fcf4ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:18.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.520+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 >> v1:172.21.15.100:6790/0 conn(0x7fcf48002860 legacy=0x7fcf48004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.520+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 >> v1:172.21.15.100:6790/0 conn(0x7fcf48002860 legacy=0x7fcf48004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:18.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf6dcfe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:18.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcf48009e50 con 0x7fcf48009110 2024-09-17T15:50:18.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2463135911 0 0) 0x7fcf48009e50 con 0x7fcf48009110 2024-09-17T15:50:18.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:50:18.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:18.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:18.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 10 monclient(hunting): my global_id is 114978 2024-09-17T15:50:18.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf480065c0 con 0x7fcf48009110 2024-09-17T15:50:18.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1090163520 0 0) 0x7fcf480065c0 con 0x7fcf48009110 2024-09-17T15:50:18.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf48009e50 con 0x7fcf48009110 2024-09-17T15:50:18.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcf5c0019f0 con 0x7fcf48009110 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 10 monclient: dump: 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:50:18.549 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:50:18.550 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:50:18.550 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:50:18.550 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:50:18.550 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:50:18.550 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:50:18.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3185979485 0 0) 0x7fcf48009e50 con 0x7fcf48009110 2024-09-17T15:50:18.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.528+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 --> v1:172.21.15.100:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcf4800c3f0 con 0x7fcf48009110 2024-09-17T15:50:18.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7667 (unknown 0 0 661510370) 0x7fcf5c006e00 con 0x7fcf48009110 2024-09-17T15:50:18.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf4ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:18.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf4ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:18.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 >> v1:172.21.15.100:6790/0 conn(0x7fcf48009110 legacy=0x7fcf480095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf4ffff640 1 -- 172.21.15.64:0/1891632472 >> v1:172.21.15.100:6790/0 conn(0x7fcf48009110 legacy=0x7fcf480095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:18.551 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:50:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:50:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 28, 2024-09-17T15:50:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 141752, 2024-09-17T15:50:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:50:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:50:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:50:18.551 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:50:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:50:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: 6 2024-09-17T15:50:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:50:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:50:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:18.552 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:50:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:50:18.554 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:50:18.555 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:50:18.556 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:50:18.556 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:50:18.556 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:50:18.556 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:50:18.556 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:50:18.556 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:50:18.556 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:50:18.556 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:50:18.556 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:50:18.556 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:50:18.556 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:50:18.556 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.556 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:50:18.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.557 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:50:18.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.557 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:50:18.557 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:50:18.557 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.557 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.557 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.557 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:50:18.557 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.559 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:50:18.559 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.559 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.559 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.559 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:50:18.559 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:50:18.559 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.559 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.559 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.559 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.562 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:50:18.563 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:50:18.565 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:50:18.566 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.566 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.566 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.566 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:50:18.566 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:50:18.566 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.566 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.566 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.566 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:50:18.567 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:50:18.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:50:18.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:50:18.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:50:18.568 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:50:18.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.568 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.568 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:50:18.568 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:50:18.568 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:50:18.568 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:50:18.568 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.569 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:18.570 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:18.570 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:50:18.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.570 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:50:18.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:50:18.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:50:18.570 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:18.570 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:50:18.570 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:50:18.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:50:18.570 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:50:18.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:50:18.570 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:50:18.571 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:50:18.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/1891632472 >> v1:172.21.15.100:6812/4210963484 conn(0x7fcf540408d0 legacy=0x7fcf54042d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf6dcfe640 10 monclient: shutdown 2024-09-17T15:50:18.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf6dcfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:18.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/1891632472 >> v1:172.21.15.64:6790/0 conn(0x7fcf680a3f50 legacy=0x7fcf681d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:18.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf67fff640 1 -- 172.21.15.64:0/1891632472 reap_dead start 2024-09-17T15:50:18.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf67fff640 1 -- 172.21.15.64:0/1891632472 reap_dead start 2024-09-17T15:50:18.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/1891632472 shutdown_connections 2024-09-17T15:50:18.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/1891632472 >> 172.21.15.64:0/1891632472 conn(0x7fcf680a53c0 msgr2=0x7fcf6809f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:18.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/1891632472 shutdown_connections 2024-09-17T15:50:18.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:50:18.529+0000 7fcf6dcfe640 1 -- 172.21.15.64:0/1891632472 wait complete. 2024-09-17T15:50:18.573 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:50:18.573 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-17T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38085]: starting. 2024-09-17T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38085]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38084 2024-09-17T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38085]: created object 0... 2024-09-17T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38085]: created object 25... 2024-09-17T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38085]: created object 49... 2024-09-17T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38085]: finishing. 2024-09-17T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38085]: shutting down. 2024-09-17T15:50:19.497 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38086]: starting. 2024-09-17T15:50:19.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38086]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38084 2024-09-17T15:50:19.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38086]: shutting down. 2024-09-17T15:50:21.593 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:21.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38121]: starting. 2024-09-17T15:50:21.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38121]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38084 2024-09-17T15:50:21.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38121]: created object 0... 2024-09-17T15:50:21.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38121]: created object 25... 2024-09-17T15:50:21.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38121]: created object 49... 2024-09-17T15:50:21.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38121]: finishing. 2024-09-17T15:50:21.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38121]: shutting down. 2024-09-17T15:50:21.605 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:21.605 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38123]: starting. 2024-09-17T15:50:21.605 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38123]: listing objects. 2024-09-17T15:50:21.605 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38123]: listed object 0... 2024-09-17T15:50:21.605 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38123]: listed object 25... 2024-09-17T15:50:21.605 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38123]: saw 50 objects 2024-09-17T15:50:21.606 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38123]: shutting down. 2024-09-17T15:50:22.522 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:22.522 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38122]: starting. 2024-09-17T15:50:22.522 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38122]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38084 2024-09-17T15:50:22.522 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38122]: shutting down. 2024-09-17T15:50:22.524 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:22.524 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:22.525 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:50:22.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:50:24.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38192]: starting. 2024-09-17T15:50:24.510 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38192]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38191 2024-09-17T15:50:24.510 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38192]: created object 0... 2024-09-17T15:50:24.510 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38192]: created object 25... 2024-09-17T15:50:24.510 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38192]: created object 49... 2024-09-17T15:50:24.510 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38192]: finishing. 2024-09-17T15:50:24.510 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38192]: shutting down. 2024-09-17T15:50:25.422 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38193]: starting. 2024-09-17T15:50:25.422 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38193]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38191 2024-09-17T15:50:25.422 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38193]: shutting down. 2024-09-17T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38228]: starting. 2024-09-17T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38228]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38191 2024-09-17T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38228]: created object 0... 2024-09-17T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38228]: created object 25... 2024-09-17T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38228]: created object 49... 2024-09-17T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38228]: finishing. 2024-09-17T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38228]: shutting down. 2024-09-17T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38230]: starting. 2024-09-17T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38230]: listing objects. 2024-09-17T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38230]: listed object 0... 2024-09-17T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38230]: listed object 25... 2024-09-17T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38230]: saw 50 objects 2024-09-17T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38230]: shutting down. 2024-09-17T15:50:28.460 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:28.460 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38229]: starting. 2024-09-17T15:50:28.461 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38229]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38191 2024-09-17T15:50:28.461 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38229]: shutting down. 2024-09-17T15:50:28.462 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:28.462 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:28.462 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:50:28.465 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:50:30.548 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38299]: starting. 2024-09-17T15:50:30.549 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38299]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38298 2024-09-17T15:50:30.549 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38299]: created object 0... 2024-09-17T15:50:30.549 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38299]: created object 25... 2024-09-17T15:50:30.549 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38299]: created object 49... 2024-09-17T15:50:30.549 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38299]: finishing. 2024-09-17T15:50:30.549 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38299]: shutting down. 2024-09-17T15:50:31.473 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38300]: starting. 2024-09-17T15:50:31.473 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38300]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38298 2024-09-17T15:50:31.474 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38300]: shutting down. 2024-09-17T15:50:33.503 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:33.503 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38335]: starting. 2024-09-17T15:50:33.503 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38335]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38298 2024-09-17T15:50:33.503 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38335]: created object 0... 2024-09-17T15:50:33.504 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38335]: created object 25... 2024-09-17T15:50:33.504 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38335]: created object 49... 2024-09-17T15:50:33.504 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38335]: finishing. 2024-09-17T15:50:33.504 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38335]: shutting down. 2024-09-17T15:50:33.514 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:33.514 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38337]: starting. 2024-09-17T15:50:33.514 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38337]: listing objects. 2024-09-17T15:50:33.514 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38337]: listed object 0... 2024-09-17T15:50:33.514 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38337]: listed object 25... 2024-09-17T15:50:33.514 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38337]: saw 50 objects 2024-09-17T15:50:33.514 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38337]: shutting down. 2024-09-17T15:50:34.435 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:34.435 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38336]: starting. 2024-09-17T15:50:34.435 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38336]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38298 2024-09-17T15:50:34.436 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38336]: shutting down. 2024-09-17T15:50:34.437 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:34.437 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:34.437 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:50:34.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:50:36.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38406]: starting. 2024-09-17T15:50:36.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38406]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38405 2024-09-17T15:50:36.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38406]: created object 0... 2024-09-17T15:50:36.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38406]: created object 25... 2024-09-17T15:50:36.594 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38406]: created object 49... 2024-09-17T15:50:36.595 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38406]: finishing. 2024-09-17T15:50:36.595 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38406]: shutting down. 2024-09-17T15:50:37.512 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38407]: starting. 2024-09-17T15:50:37.512 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38407]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38405 2024-09-17T15:50:37.512 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38407]: shutting down. 2024-09-17T15:50:39.609 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:39.610 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38442]: starting. 2024-09-17T15:50:39.610 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38442]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38405 2024-09-17T15:50:39.610 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38442]: created object 0... 2024-09-17T15:50:39.610 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38442]: created object 25... 2024-09-17T15:50:39.610 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38442]: created object 49... 2024-09-17T15:50:39.610 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38442]: finishing. 2024-09-17T15:50:39.610 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38442]: shutting down. 2024-09-17T15:50:39.620 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:39.620 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38444]: starting. 2024-09-17T15:50:39.620 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38444]: listing objects. 2024-09-17T15:50:39.620 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38444]: listed object 0... 2024-09-17T15:50:39.621 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38444]: listed object 25... 2024-09-17T15:50:39.621 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38444]: saw 50 objects 2024-09-17T15:50:39.621 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38444]: shutting down. 2024-09-17T15:50:41.545 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:41.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38443]: starting. 2024-09-17T15:50:41.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38443]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38405 2024-09-17T15:50:41.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38443]: shutting down. 2024-09-17T15:50:41.546 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:41.546 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:41.546 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:50:41.549 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:50:43.642 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38513]: starting. 2024-09-17T15:50:43.642 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38513]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38512 2024-09-17T15:50:43.642 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38513]: created object 0... 2024-09-17T15:50:43.643 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38513]: created object 25... 2024-09-17T15:50:43.643 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38513]: created object 49... 2024-09-17T15:50:43.643 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38513]: finishing. 2024-09-17T15:50:43.643 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38513]: shutting down. 2024-09-17T15:50:44.560 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38514]: starting. 2024-09-17T15:50:44.560 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38514]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38512 2024-09-17T15:50:44.560 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38514]: shutting down. 2024-09-17T15:50:46.685 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:46.685 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38549]: starting. 2024-09-17T15:50:46.685 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38549]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38512 2024-09-17T15:50:46.685 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38549]: created object 0... 2024-09-17T15:50:46.685 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38549]: created object 25... 2024-09-17T15:50:46.686 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38549]: created object 49... 2024-09-17T15:50:46.686 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38549]: finishing. 2024-09-17T15:50:46.686 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38549]: shutting down. 2024-09-17T15:50:46.696 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:46.696 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38551]: starting. 2024-09-17T15:50:46.696 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38551]: listing objects. 2024-09-17T15:50:46.697 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38551]: listed object 0... 2024-09-17T15:50:46.697 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38551]: listed object 25... 2024-09-17T15:50:46.697 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38551]: saw 50 objects 2024-09-17T15:50:46.697 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38551]: shutting down. 2024-09-17T15:50:47.593 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:47.593 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38550]: starting. 2024-09-17T15:50:47.593 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38550]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38512 2024-09-17T15:50:47.593 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38550]: shutting down. 2024-09-17T15:50:47.595 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:47.596 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:47.596 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:50:47.599 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:50:49.693 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38620]: starting. 2024-09-17T15:50:49.693 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38620]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38619 2024-09-17T15:50:49.693 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38620]: created object 0... 2024-09-17T15:50:49.693 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38620]: created object 25... 2024-09-17T15:50:49.693 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38620]: created object 49... 2024-09-17T15:50:49.693 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38620]: finishing. 2024-09-17T15:50:49.693 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38620]: shutting down. 2024-09-17T15:50:50.624 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38621]: starting. 2024-09-17T15:50:50.624 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38621]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38619 2024-09-17T15:50:50.624 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38621]: shutting down. 2024-09-17T15:50:52.718 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:52.719 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38656]: starting. 2024-09-17T15:50:52.719 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38656]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38619 2024-09-17T15:50:52.719 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38656]: created object 0... 2024-09-17T15:50:52.719 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38656]: created object 25... 2024-09-17T15:50:52.719 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38656]: created object 49... 2024-09-17T15:50:52.719 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38656]: finishing. 2024-09-17T15:50:52.719 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38656]: shutting down. 2024-09-17T15:50:52.729 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:52.730 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38658]: starting. 2024-09-17T15:50:52.730 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38658]: listing objects. 2024-09-17T15:50:52.730 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38658]: listed object 0... 2024-09-17T15:50:52.730 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38658]: listed object 25... 2024-09-17T15:50:52.730 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38658]: saw 50 objects 2024-09-17T15:50:52.730 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38658]: shutting down. 2024-09-17T15:50:53.426 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:53.427 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38657]: starting. 2024-09-17T15:50:53.427 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38657]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38619 2024-09-17T15:50:53.427 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38657]: shutting down. 2024-09-17T15:50:53.428 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:53.428 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:53.428 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:50:53.431 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:50:55.529 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38727]: starting. 2024-09-17T15:50:55.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38727]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38726 2024-09-17T15:50:55.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38727]: created object 0... 2024-09-17T15:50:55.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38727]: created object 25... 2024-09-17T15:50:55.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38727]: created object 49... 2024-09-17T15:50:55.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38727]: finishing. 2024-09-17T15:50:55.530 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38727]: shutting down. 2024-09-17T15:50:56.473 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38728]: starting. 2024-09-17T15:50:56.473 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38728]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38726 2024-09-17T15:50:56.473 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38728]: shutting down. 2024-09-17T15:50:58.508 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:58.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38764]: starting. 2024-09-17T15:50:58.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38764]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38726 2024-09-17T15:50:58.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38764]: created object 0... 2024-09-17T15:50:58.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38764]: created object 25... 2024-09-17T15:50:58.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38764]: created object 49... 2024-09-17T15:50:58.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38764]: finishing. 2024-09-17T15:50:58.509 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38764]: shutting down. 2024-09-17T15:50:58.519 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:58.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38766]: starting. 2024-09-17T15:50:58.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38766]: listing objects. 2024-09-17T15:50:58.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38766]: listed object 0... 2024-09-17T15:50:58.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38766]: listed object 25... 2024-09-17T15:50:58.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38766]: saw 50 objects 2024-09-17T15:50:58.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38766]: shutting down. 2024-09-17T15:50:59.427 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:59.427 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38765]: starting. 2024-09-17T15:50:59.427 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38765]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38726 2024-09-17T15:50:59.427 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38765]: shutting down. 2024-09-17T15:50:59.429 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:59.429 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:50:59.429 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:50:59.433 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:51:01.584 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38836]: starting. 2024-09-17T15:51:01.584 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38836]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38835 2024-09-17T15:51:01.584 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38836]: created object 0... 2024-09-17T15:51:01.585 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38836]: created object 25... 2024-09-17T15:51:01.585 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38836]: created object 49... 2024-09-17T15:51:01.585 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38836]: finishing. 2024-09-17T15:51:01.585 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38836]: shutting down. 2024-09-17T15:51:02.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38837]: starting. 2024-09-17T15:51:02.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38837]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38835 2024-09-17T15:51:02.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38837]: shutting down. 2024-09-17T15:51:04.590 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:04.590 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38872]: starting. 2024-09-17T15:51:04.590 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38872]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38835 2024-09-17T15:51:04.590 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38872]: created object 0... 2024-09-17T15:51:04.590 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38872]: created object 25... 2024-09-17T15:51:04.590 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38872]: created object 49... 2024-09-17T15:51:04.590 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38872]: finishing. 2024-09-17T15:51:04.590 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38872]: shutting down. 2024-09-17T15:51:04.600 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:04.600 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38874]: starting. 2024-09-17T15:51:04.600 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38874]: listing objects. 2024-09-17T15:51:04.601 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38874]: listed object 0... 2024-09-17T15:51:04.601 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38874]: listed object 25... 2024-09-17T15:51:04.601 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38874]: saw 50 objects 2024-09-17T15:51:04.601 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38874]: shutting down. 2024-09-17T15:51:05.515 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:05.516 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38873]: starting. 2024-09-17T15:51:05.516 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38873]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38835 2024-09-17T15:51:05.516 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38873]: shutting down. 2024-09-17T15:51:05.517 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:05.517 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:05.518 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:51:05.521 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:51:07.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38943]: starting. 2024-09-17T15:51:07.614 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38943]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38942 2024-09-17T15:51:07.614 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38943]: created object 0... 2024-09-17T15:51:07.614 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38943]: created object 25... 2024-09-17T15:51:07.614 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38943]: created object 49... 2024-09-17T15:51:07.614 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38943]: finishing. 2024-09-17T15:51:07.614 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[38943]: shutting down. 2024-09-17T15:51:08.429 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38944]: starting. 2024-09-17T15:51:08.430 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38944]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38942 2024-09-17T15:51:08.430 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[38944]: shutting down. 2024-09-17T15:51:10.519 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:10.520 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38979]: starting. 2024-09-17T15:51:10.520 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38979]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-38942 2024-09-17T15:51:10.520 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38979]: created object 0... 2024-09-17T15:51:10.520 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38979]: created object 25... 2024-09-17T15:51:10.520 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38979]: created object 49... 2024-09-17T15:51:10.520 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38979]: finishing. 2024-09-17T15:51:10.520 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[38979]: shutting down. 2024-09-17T15:51:10.532 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:10.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38981]: starting. 2024-09-17T15:51:10.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38981]: listing objects. 2024-09-17T15:51:10.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38981]: listed object 0... 2024-09-17T15:51:10.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38981]: listed object 25... 2024-09-17T15:51:10.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38981]: saw 50 objects 2024-09-17T15:51:10.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[38981]: shutting down. 2024-09-17T15:51:11.436 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:11.436 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38980]: starting. 2024-09-17T15:51:11.436 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38980]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-38942 2024-09-17T15:51:11.437 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[38980]: shutting down. 2024-09-17T15:51:11.438 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:11.439 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:11.439 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:51:11.442 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:51:13.543 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39050]: starting. 2024-09-17T15:51:13.543 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39050]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39049 2024-09-17T15:51:13.543 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39050]: created object 0... 2024-09-17T15:51:13.543 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39050]: created object 25... 2024-09-17T15:51:13.543 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39050]: created object 49... 2024-09-17T15:51:13.543 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39050]: finishing. 2024-09-17T15:51:13.543 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39050]: shutting down. 2024-09-17T15:51:14.461 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39051]: starting. 2024-09-17T15:51:14.461 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39051]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39049 2024-09-17T15:51:14.461 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39051]: shutting down. 2024-09-17T15:51:16.574 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:16.574 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39086]: starting. 2024-09-17T15:51:16.574 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39086]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39049 2024-09-17T15:51:16.574 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39086]: created object 0... 2024-09-17T15:51:16.575 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39086]: created object 25... 2024-09-17T15:51:16.575 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39086]: created object 49... 2024-09-17T15:51:16.575 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39086]: finishing. 2024-09-17T15:51:16.575 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39086]: shutting down. 2024-09-17T15:51:17.497 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:17.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39088]: starting. 2024-09-17T15:51:17.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39088]: listing objects. 2024-09-17T15:51:17.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39088]: listed object 0... 2024-09-17T15:51:17.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39088]: listed object 25... 2024-09-17T15:51:17.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39088]: saw 50 objects 2024-09-17T15:51:17.498 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39088]: shutting down. 2024-09-17T15:51:18.434 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:18.434 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39087]: starting. 2024-09-17T15:51:18.434 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39087]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39049 2024-09-17T15:51:18.434 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39087]: shutting down. 2024-09-17T15:51:18.435 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:18.435 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:18.435 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:51:18.438 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:51:20.517 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39157]: starting. 2024-09-17T15:51:20.517 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39157]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39156 2024-09-17T15:51:20.517 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39157]: created object 0... 2024-09-17T15:51:20.517 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39157]: created object 25... 2024-09-17T15:51:20.517 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39157]: created object 49... 2024-09-17T15:51:20.518 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39157]: finishing. 2024-09-17T15:51:20.518 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39157]: shutting down. 2024-09-17T15:51:21.438 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39158]: starting. 2024-09-17T15:51:21.438 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39158]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39156 2024-09-17T15:51:21.438 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39158]: shutting down. 2024-09-17T15:51:23.545 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:23.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39193]: starting. 2024-09-17T15:51:23.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39193]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39156 2024-09-17T15:51:23.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39193]: created object 0... 2024-09-17T15:51:23.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39193]: created object 25... 2024-09-17T15:51:23.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39193]: created object 49... 2024-09-17T15:51:23.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39193]: finishing. 2024-09-17T15:51:23.545 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39193]: shutting down. 2024-09-17T15:51:23.555 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:23.555 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39195]: starting. 2024-09-17T15:51:23.555 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39195]: listing objects. 2024-09-17T15:51:23.555 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39195]: listed object 0... 2024-09-17T15:51:23.556 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39195]: listed object 25... 2024-09-17T15:51:23.556 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39195]: saw 50 objects 2024-09-17T15:51:23.556 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39195]: shutting down. 2024-09-17T15:51:24.469 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:24.470 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39194]: starting. 2024-09-17T15:51:24.470 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39194]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39156 2024-09-17T15:51:24.470 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39194]: shutting down. 2024-09-17T15:51:24.471 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:24.471 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:24.471 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:51:24.474 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:51:26.557 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39264]: starting. 2024-09-17T15:51:26.557 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39264]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39263 2024-09-17T15:51:26.557 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39264]: created object 0... 2024-09-17T15:51:26.558 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39264]: created object 25... 2024-09-17T15:51:26.558 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39264]: created object 49... 2024-09-17T15:51:26.558 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39264]: finishing. 2024-09-17T15:51:26.558 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39264]: shutting down. 2024-09-17T15:51:27.481 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39265]: starting. 2024-09-17T15:51:27.481 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39265]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39263 2024-09-17T15:51:27.481 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39265]: shutting down. 2024-09-17T15:51:29.518 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:29.518 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39300]: starting. 2024-09-17T15:51:29.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39300]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39263 2024-09-17T15:51:29.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39300]: created object 0... 2024-09-17T15:51:29.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39300]: created object 25... 2024-09-17T15:51:29.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39300]: created object 49... 2024-09-17T15:51:29.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39300]: finishing. 2024-09-17T15:51:29.519 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39300]: shutting down. 2024-09-17T15:51:29.531 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:29.531 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39302]: starting. 2024-09-17T15:51:29.531 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39302]: listing objects. 2024-09-17T15:51:29.531 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39302]: listed object 0... 2024-09-17T15:51:29.531 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39302]: listed object 25... 2024-09-17T15:51:29.531 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39302]: saw 50 objects 2024-09-17T15:51:29.531 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39302]: shutting down. 2024-09-17T15:51:30.450 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:30.450 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39301]: starting. 2024-09-17T15:51:30.451 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39301]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39263 2024-09-17T15:51:30.451 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39301]: shutting down. 2024-09-17T15:51:30.452 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:30.452 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:30.452 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:51:30.454 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:51:32.551 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39371]: starting. 2024-09-17T15:51:32.551 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39371]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39370 2024-09-17T15:51:32.551 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39371]: created object 0... 2024-09-17T15:51:32.551 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39371]: created object 25... 2024-09-17T15:51:32.552 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39371]: created object 49... 2024-09-17T15:51:32.552 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39371]: finishing. 2024-09-17T15:51:32.552 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39371]: shutting down. 2024-09-17T15:51:33.471 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39372]: starting. 2024-09-17T15:51:33.471 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39372]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39370 2024-09-17T15:51:33.471 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39372]: shutting down. 2024-09-17T15:51:35.578 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:35.578 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39407]: starting. 2024-09-17T15:51:35.578 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39407]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39370 2024-09-17T15:51:35.578 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39407]: created object 0... 2024-09-17T15:51:35.578 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39407]: created object 25... 2024-09-17T15:51:35.578 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39407]: created object 49... 2024-09-17T15:51:35.578 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39407]: finishing. 2024-09-17T15:51:35.578 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39407]: shutting down. 2024-09-17T15:51:35.588 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:35.588 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39409]: starting. 2024-09-17T15:51:35.588 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39409]: listing objects. 2024-09-17T15:51:35.589 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39409]: listed object 0... 2024-09-17T15:51:35.589 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39409]: listed object 25... 2024-09-17T15:51:35.589 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39409]: saw 50 objects 2024-09-17T15:51:35.589 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39409]: shutting down. 2024-09-17T15:51:36.505 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39408]: starting. 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39408]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39370 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39408]: shutting down. 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:51:36.509 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:51:38.527 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39478]: starting. 2024-09-17T15:51:38.527 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39478]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39477 2024-09-17T15:51:38.527 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39478]: created object 0... 2024-09-17T15:51:38.527 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39478]: created object 25... 2024-09-17T15:51:38.527 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39478]: created object 49... 2024-09-17T15:51:38.527 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39478]: finishing. 2024-09-17T15:51:38.528 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39478]: shutting down. 2024-09-17T15:51:39.438 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39479]: starting. 2024-09-17T15:51:39.439 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39479]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39477 2024-09-17T15:51:39.439 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39479]: shutting down. 2024-09-17T15:51:43.665 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:43.665 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39514]: starting. 2024-09-17T15:51:43.665 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39514]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39477 2024-09-17T15:51:43.665 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39514]: created object 0... 2024-09-17T15:51:43.666 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39514]: created object 25... 2024-09-17T15:51:43.666 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39514]: created object 49... 2024-09-17T15:51:43.666 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39514]: finishing. 2024-09-17T15:51:43.666 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39514]: shutting down. 2024-09-17T15:51:43.676 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:43.676 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39516]: starting. 2024-09-17T15:51:43.676 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39516]: listing objects. 2024-09-17T15:51:43.676 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39516]: listed object 0... 2024-09-17T15:51:43.676 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39516]: listed object 25... 2024-09-17T15:51:43.676 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39516]: saw 50 objects 2024-09-17T15:51:43.676 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39516]: shutting down. 2024-09-17T15:51:44.583 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:44.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39515]: starting. 2024-09-17T15:51:44.583 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39515]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39477 2024-09-17T15:51:44.584 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39515]: shutting down. 2024-09-17T15:51:44.585 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:44.585 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:44.585 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:51:44.588 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:51:48.524 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39585]: starting. 2024-09-17T15:51:48.524 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39585]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39584 2024-09-17T15:51:48.524 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39585]: created object 0... 2024-09-17T15:51:48.524 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39585]: created object 25... 2024-09-17T15:51:48.524 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39585]: created object 49... 2024-09-17T15:51:48.524 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39585]: finishing. 2024-09-17T15:51:48.524 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39585]: shutting down. 2024-09-17T15:51:48.574 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.i 2024-09-17T15:51:48.574 INFO:tasks.ceph.mon.i:Restarting daemon 2024-09-17T15:51:48.575 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-09-17T15:51:48.577 INFO:tasks.ceph.mon.i:Started 2024-09-17T15:51:48.577 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-09-17T15:51:48.577 INFO:tasks.ceph.mon.e:Restarting daemon 2024-09-17T15:51:48.577 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-09-17T15:51:48.580 INFO:tasks.ceph.mon.e:Started 2024-09-17T15:51:48.580 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.h 2024-09-17T15:51:48.580 INFO:tasks.ceph.mon.h:Restarting daemon 2024-09-17T15:51:48.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-09-17T15:51:48.583 INFO:tasks.ceph.mon.h:Started 2024-09-17T15:51:48.583 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-17T15:51:48.583 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T15:51:48.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T15:51:48.585 INFO:tasks.ceph.mon.c:Started 2024-09-17T15:51:48.585 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-17T15:51:48.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:51:48.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.693+0000 7fb941364640 10 monclient: get_monmap_and_config 2024-09-17T15:51:48.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.693+0000 7fb941364640 10 monclient: build_initial_monmap 2024-09-17T15:51:48.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.694+0000 7fb941364640 10 monclient: monmap: 2024-09-17T15:51:48.696 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:48.696 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:48.696 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:48.694881+0000 2024-09-17T15:51:48.696 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:48.694881+0000 2024-09-17T15:51:48.696 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:48.696 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:48.696 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:48.696 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:48.696 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:48.697 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:48.697 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:48.697 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:48.697 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:48.697 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:48.697 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:48.697 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:48.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.694+0000 7fb941364640 1 Processor -- start 2024-09-17T15:51:48.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.694+0000 7fb941364640 1 -- start start 2024-09-17T15:51:48.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.694+0000 7fb941364640 10 monclient: init 2024-09-17T15:51:48.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.695+0000 7fb941364640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:48.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.695+0000 7fb941364640 10 monclient: _add_conns ranks=[6,7,3,4,1,0,2,8,5] 2024-09-17T15:51:48.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.695+0000 7fb941364640 10 monclient(hunting): picked mon.noname-d con 0x7fb93c0a54b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:51:48.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.695+0000 7fb941364640 10 monclient(hunting): picked mon.noname-i con 0x7fb93c135c80 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:48.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.695+0000 7fb941364640 10 monclient(hunting): picked mon.noname-g con 0x7fb93c1360a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:48.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.695+0000 7fb941364640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb93c0aacb0 con 0x7fb93c1360a0 2024-09-17T15:51:48.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.695+0000 7fb93affd640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fb93c0a54b0 0x7fb93c0a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:51414/0 (socket says 172.21.15.64:51414) 2024-09-17T15:51:48.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.695+0000 7fb93affd640 1 -- 172.21.15.64:0/620010651 learned_addr learned my addr 172.21.15.64:0/620010651 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:48.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.695+0000 7fb941364640 1 -- 172.21.15.64:0/620010651 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb93c09d780 con 0x7fb93c0a54b0 2024-09-17T15:51:48.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.695+0000 7fb941364640 1 -- 172.21.15.64:0/620010651 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb93c09da60 con 0x7fb93c135c80 2024-09-17T15:51:48.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.695+0000 7fb941364640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:48.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:48.695+0000 7fb941364640 10 monclient(hunting): authenticate will time out at 1106.370790s 2024-09-17T15:51:51.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 10 monclient(hunting): tick 2024-09-17T15:51:51.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:51:51.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 1 monclient(hunting): continuing hunt 2024-09-17T15:51:51.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:51:51.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 1 -- 172.21.15.64:0/620010651 >> v1:172.21.15.100:6792/0 conn(0x7fb93c135c80 legacy=0x7fb93c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:51.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 1 -- 172.21.15.64:0/620010651 >> v1:172.21.15.64:6792/0 conn(0x7fb93c0a54b0 legacy=0x7fb93c0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:51.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 1 -- 172.21.15.64:0/620010651 >> v1:172.21.15.100:6790/0 conn(0x7fb93c1360a0 legacy=0x7fb93c141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:51.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 10 monclient: _add_conns ranks=[8,1,0,2,5,4] 2024-09-17T15:51:51.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 10 monclient(hunting): picked mon.noname-e con 0x7fb91c0037e0 addr v1:172.21.15.64:6793/0 2024-09-17T15:51:51.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 10 monclient(hunting): picked mon.noname-f con 0x7fb91c007470 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:51.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb91c00b200 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:51.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 1 -- 172.21.15.64:0/620010651 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb93c09dd40 con 0x7fb91c00b200 2024-09-17T15:51:51.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 1 -- 172.21.15.64:0/620010651 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb91c010020 con 0x7fb91c007470 2024-09-17T15:51:51.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 1 -- 172.21.15.64:0/620010651 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb91c0112a0 con 0x7fb91c0037e0 2024-09-17T15:51:51.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:51.695+0000 7fb9397fa640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.705+0000 7fb93a7fc640 1 -- 172.21.15.64:0/620010651 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3675373029 0 0) 0x7fb93c09dd40 con 0x7fb91c00b200 2024-09-17T15:51:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.705+0000 7fb93a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.705+0000 7fb93a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.705+0000 7fb93a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.705+0000 7fb93a7fc640 10 monclient(hunting): my global_id is 124686 2024-09-17T15:51:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.705+0000 7fb93a7fc640 1 -- 172.21.15.64:0/620010651 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb928003520 con 0x7fb91c00b200 2024-09-17T15:51:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.705+0000 7fb93a7fc640 1 -- 172.21.15.64:0/620010651 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3539239205 0 0) 0x7fb91c010020 con 0x7fb91c007470 2024-09-17T15:51:53.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.705+0000 7fb93a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:53.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.705+0000 7fb93a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.705+0000 7fb93a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.705+0000 7fb93a7fc640 10 monclient(hunting): my global_id is 6517 2024-09-17T15:51:53.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.705+0000 7fb93a7fc640 1 -- 172.21.15.64:0/620010651 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb93c09dd40 con 0x7fb91c007470 2024-09-17T15:51:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 1 -- 172.21.15.64:0/620010651 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3109204231 0 0) 0x7fb93c09dd40 con 0x7fb91c007470 2024-09-17T15:51:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 1 -- 172.21.15.64:0/620010651 >> v1:172.21.15.64:6793/0 conn(0x7fb91c0037e0 legacy=0x7fb91c005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 1 -- 172.21.15.64:0/620010651 >> v1:172.21.15.64:6789/0 conn(0x7fb91c00b200 legacy=0x7fb91c00d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:51:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 1 monclient: found mon.noname-f 2024-09-17T15:51:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.100:6789/0 2024-09-17T15:51:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93b7fe640 1 -- 172.21.15.64:0/620010651 reap_dead start 2024-09-17T15:51:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 1 -- 172.21.15.64:0/620010651 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91c012580 con 0x7fb91c007470 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 1 -- 172.21.15.64:0/620010651 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb924001830 con 0x7fb91c007470 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 10 monclient: dump: 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb941364640 5 monclient: authenticate success, global_id 6517 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb941364640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 1 -- 172.21.15.64:0/620010651 <== mon.1 v1:172.21.15.100:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb924001a40 con 0x7fb91c007470 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 1 -- 172.21.15.64:0/620010651 <== mon.1 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb924003930 con 0x7fb91c007470 2024-09-17T15:51:53.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:51:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb93a7fc640 10 monclient: dump: 2024-09-17T15:51:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:53.713 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb941364640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.707+0000 7fb941364640 4 set_mon_vals no callback set 2024-09-17T15:51:53.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.708+0000 7fb941364640 10 monclient: shutdown 2024-09-17T15:51:53.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.708+0000 7fb941364640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:53.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.708+0000 7fb941364640 1 -- 172.21.15.64:0/620010651 >> v1:172.21.15.100:6789/0 conn(0x7fb91c007470 legacy=0x7fb91c009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.708+0000 7fb941364640 1 -- 172.21.15.64:0/620010651 shutdown_connections 2024-09-17T15:51:53.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.708+0000 7fb941364640 1 -- 172.21.15.64:0/620010651 >> 172.21.15.64:0/620010651 conn(0x7fb93c06daa0 msgr2=0x7fb93c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:53.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.708+0000 7fb941364640 1 -- 172.21.15.64:0/620010651 shutdown_connections 2024-09-17T15:51:53.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.708+0000 7fb941364640 1 -- 172.21.15.64:0/620010651 wait complete. 2024-09-17T15:51:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.709+0000 7fb941364640 10 monclient: build_initial_monmap 2024-09-17T15:51:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.709+0000 7fb941364640 10 monclient: monmap: 2024-09-17T15:51:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:53.716 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.710+0000 7fb941364640 1 Processor -- start 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.710+0000 7fb941364640 1 -- start start 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.710+0000 7fb941364640 10 monclient: init 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.711+0000 7fb941364640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.711+0000 7fb941364640 10 monclient: _add_conns ranks=[3,7,5,0,6,1,2,4,8] 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.711+0000 7fb941364640 10 monclient(hunting): picked mon.noname-g con 0x7fb91c007470 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.711+0000 7fb941364640 10 monclient(hunting): picked mon.noname-i con 0x7fb93c1ce4d0 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.711+0000 7fb941364640 10 monclient(hunting): picked mon.noname-h con 0x7fb93c1ce8f0 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.711+0000 7fb941364640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb93c0a3b30 con 0x7fb91c007470 2024-09-17T15:51:53.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.711+0000 7fb941364640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb93c0a2640 con 0x7fb93c1ce8f0 2024-09-17T15:51:53.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.711+0000 7fb941364640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb93c0a49a0 con 0x7fb93c1ce4d0 2024-09-17T15:51:53.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.711+0000 7fb941364640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:53.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.711+0000 7fb941364640 10 monclient(hunting): authenticate will time out at 1111.386609s 2024-09-17T15:51:53.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.711+0000 7fb93affd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fb91c007470 0x7fb93c1cddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:48004/0 (socket says 172.21.15.64:48004) 2024-09-17T15:51:53.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.711+0000 7fb93affd640 1 -- 172.21.15.64:0/4107619537 learned_addr learned my addr 172.21.15.64:0/4107619537 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:53.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.711+0000 7fb93b7fe640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7fb93c1ce8f0 0x7fb93c1d6730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:49730/0 (socket says 172.21.15.64:49730) 2024-09-17T15:51:53.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.714+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1575817017 0 0) 0x7fb93c0a3b30 con 0x7fb91c007470 2024-09-17T15:51:53.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.714+0000 7fb9337fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:53.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.714+0000 7fb9337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.714+0000 7fb9337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.714+0000 7fb9337fe640 10 monclient(hunting): my global_id is 115410 2024-09-17T15:51:53.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.714+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb91c011550 con 0x7fb91c007470 2024-09-17T15:51:53.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 534194620 0 0) 0x7fb93c0a49a0 con 0x7fb93c1ce4d0 2024-09-17T15:51:53.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:53.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 10 monclient(hunting): my global_id is 154105 2024-09-17T15:51:53.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb93c0a3b30 con 0x7fb93c1ce4d0 2024-09-17T15:51:53.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3702507952 0 0) 0x7fb91c011550 con 0x7fb91c007470 2024-09-17T15:51:53.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb93c0a49a0 con 0x7fb91c007470 2024-09-17T15:51:53.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb92c001a50 con 0x7fb91c007470 2024-09-17T15:51:53.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:53.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:51:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 10 monclient(hunting): dump: 2024-09-17T15:51:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:53.727 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1150285557 0 0) 0x7fb93c0a3b30 con 0x7fb93c1ce4d0 2024-09-17T15:51:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.720+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb91c011550 con 0x7fb93c1ce4d0 2024-09-17T15:51:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb924001890 con 0x7fb93c1ce4d0 2024-09-17T15:51:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:51:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 10 monclient(hunting): dump: 2024-09-17T15:51:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 687477651 0 0) 0x7fb93c0a49a0 con 0x7fb91c007470 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 >> v1:172.21.15.100:6792/0 conn(0x7fb93c1ce4d0 legacy=0x7fb93c1d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 >> v1:172.21.15.100:6791/0 conn(0x7fb93c1ce8f0 legacy=0x7fb93c1d6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 10 monclient: _finish_hunting 0 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 1 monclient: found mon.g 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:51:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb93c132e50 con 0x7fb91c007470 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb941364640 5 monclient: authenticate success, global_id 115410 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb941364640 10 monclient: _renew_subs 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb941364640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb941364640 1 -- 172.21.15.64:0/4107619537 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb93c0a1b60 con 0x7fb91c007470 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb941364640 10 monclient: _renew_subs 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb941364640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 <== mon.3 v1:172.21.15.100:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb92c001cf0 con 0x7fb91c007470 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb941364640 1 -- 172.21.15.64:0/4107619537 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb93c1d7e50 con 0x7fb91c007470 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 <== mon.3 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb92c004dd0 con 0x7fb91c007470 2024-09-17T15:51:53.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb939ffb640 4 set_mon_vals no callback set 2024-09-17T15:51:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:51:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.721+0000 7fb9337fe640 10 monclient: dump: 2024-09-17T15:51:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:53.732 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.722+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 <== mon.3 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fb92c004f70 con 0x7fb91c007470 2024-09-17T15:51:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.722+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 <== mon.3 v1:172.21.15.100:6790/0 8 ==== osd_map(284..284 src has 192..284) ==== 3898+0+0 (unknown 1175997477 0 0) 0x7fb92c04ec40 con 0x7fb91c007470 2024-09-17T15:51:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.723+0000 7fb941364640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.723+0000 7fb941364640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.723+0000 7fb941364640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:51:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.723+0000 7fb941364640 1 -- 172.21.15.64:0/4107619537 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb920005080 con 0x7fb91c007470 2024-09-17T15:51:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.723+0000 7fb933fff640 10 monclient: get_auth_request con 0x7fb91c041f60 auth_method 0 2024-09-17T15:51:53.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.729+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 <== mon.3 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fb92c0177d0 con 0x7fb91c007470 2024-09-17T15:51:53.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.729+0000 7fb9337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:53.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.729+0000 7fb9337fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:53.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.988+0000 7fb941364640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:51:53.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.988+0000 7fb941364640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:53.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.988+0000 7fb941364640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:51:53.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.988+0000 7fb941364640 1 -- 172.21.15.64:0/4107619537 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb920005370 con 0x7fb91c007470 2024-09-17T15:51:53.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.989+0000 7fb9337fe640 1 -- 172.21.15.64:0/4107619537 <== mon.3 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 3501320295) 0x7fb92c0170d0 con 0x7fb91c007470 2024-09-17T15:51:53.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.989+0000 7fb9337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:53.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.989+0000 7fb9337fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:53.992 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":32,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:51:53.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.991+0000 7fb941364640 1 -- 172.21.15.64:0/4107619537 >> v1:172.21.15.100:6812/4210963484 conn(0x7fb91c041f60 legacy=0x7fb91c044420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.991+0000 7fb941364640 10 monclient: shutdown 2024-09-17T15:51:53.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.992+0000 7fb941364640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:53.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.992+0000 7fb941364640 1 -- 172.21.15.64:0/4107619537 >> v1:172.21.15.100:6790/0 conn(0x7fb91c007470 legacy=0x7fb93c1cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.992+0000 7fb941364640 1 -- 172.21.15.64:0/4107619537 shutdown_connections 2024-09-17T15:51:53.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.992+0000 7fb941364640 1 -- 172.21.15.64:0/4107619537 >> 172.21.15.64:0/4107619537 conn(0x7fb93c06daa0 msgr2=0x7fb93c1bc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:53.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.992+0000 7fb941364640 1 -- 172.21.15.64:0/4107619537 shutdown_connections 2024-09-17T15:51:53.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:53.992+0000 7fb941364640 1 -- 172.21.15.64:0/4107619537 wait complete. 2024-09-17T15:51:54.004 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-17T15:51:54.004 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-17T15:51:54.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:51:54.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.104+0000 7f893e6b8640 10 monclient: get_monmap_and_config 2024-09-17T15:51:54.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.104+0000 7f893e6b8640 10 monclient: build_initial_monmap 2024-09-17T15:51:54.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.104+0000 7f893e6b8640 10 monclient: monmap: 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:54.105692+0000 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:54.105692+0000 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:54.108 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:54.108 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.105+0000 7f893e6b8640 1 Processor -- start 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.105+0000 7f893e6b8640 1 -- start start 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.105+0000 7f893e6b8640 10 monclient: init 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893e6b8640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893e6b8640 10 monclient: _add_conns ranks=[3,5,4,0,2,7,1,6,8] 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893e6b8640 10 monclient(hunting): picked mon.noname-g con 0x7f89380a5520 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893e6b8640 10 monclient(hunting): picked mon.noname-h con 0x7f8938135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893e6b8640 10 monclient(hunting): picked mon.noname-c con 0x7f8938135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893e6b8640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f893809d6f0 con 0x7f89380a5520 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893e6b8640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f893809d9d0 con 0x7f8938135f70 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893e6b8640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f893809dcb0 con 0x7f8938135b50 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893e6b8640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893e6b8640 10 monclient(hunting): authenticate will time out at 1111.781223s 2024-09-17T15:51:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893cc55640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f8938135f70 0x7f89381415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:46802/0 (socket says 172.21.15.64:46802) 2024-09-17T15:51:54.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893cc55640 1 -- 172.21.15.64:0/3712490980 learned_addr learned my addr 172.21.15.64:0/3712490980 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:54.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 1 -- 172.21.15.64:0/3712490980 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3661687252 0 0) 0x7f893809d9d0 con 0x7f8938135f70 2024-09-17T15:51:54.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient(hunting): my global_id is 154102 2024-09-17T15:51:54.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 1 -- 172.21.15.64:0/3712490980 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8920003520 con 0x7f8938135f70 2024-09-17T15:51:54.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 1 -- 172.21.15.64:0/3712490980 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2015707348 0 0) 0x7f893809d6f0 con 0x7f89380a5520 2024-09-17T15:51:54.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient(hunting): my global_id is 115419 2024-09-17T15:51:54.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 1 -- 172.21.15.64:0/3712490980 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f893809d9d0 con 0x7f89380a5520 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 1 -- 172.21.15.64:0/3712490980 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 878399365 0 0) 0x7f8920003520 con 0x7f8938135f70 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 1 -- 172.21.15.64:0/3712490980 >> v1:172.21.15.100:6791/0 conn(0x7f8938135b50 legacy=0x7f893813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 1 -- 172.21.15.64:0/3712490980 >> v1:172.21.15.100:6790/0 conn(0x7f89380a5520 legacy=0x7f89380a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 1 monclient: found mon.noname-c 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 1 -- 172.21.15.64:0/3712490980 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893809ccd0 con 0x7f8938135f70 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient: _finish_auth 0 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 1 -- 172.21.15.64:0/3712490980 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8930001b30 con 0x7f8938135f70 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893e6b8640 5 monclient: authenticate success, global_id 154102 2024-09-17T15:51:54.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893e6b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 10 monclient: dump: 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.119 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f893e6b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.106+0000 7f8936ffd640 1 -- 172.21.15.64:0/3712490980 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8930001cd0 con 0x7f8938135f70 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f8936ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f8936ffd640 1 -- 172.21.15.64:0/3712490980 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8930004f60 con 0x7f8938135f70 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f8936ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f8936ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f8936ffd640 10 monclient: dump: 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.120 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f893e6b8640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f893e6b8640 4 set_mon_vals no callback set 2024-09-17T15:51:54.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f893e6b8640 10 monclient: shutdown 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f893e6b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f893e6b8640 1 -- 172.21.15.64:0/3712490980 >> v1:172.21.15.64:6791/0 conn(0x7f8938135f70 legacy=0x7f89381415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f893e6b8640 1 -- 172.21.15.64:0/3712490980 shutdown_connections 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f893e6b8640 1 -- 172.21.15.64:0/3712490980 >> 172.21.15.64:0/3712490980 conn(0x7f893806db30 msgr2=0x7f89380ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f893e6b8640 1 -- 172.21.15.64:0/3712490980 shutdown_connections 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.107+0000 7f893e6b8640 1 -- 172.21.15.64:0/3712490980 wait complete. 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.108+0000 7f893e6b8640 10 monclient: build_initial_monmap 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.108+0000 7f893e6b8640 10 monclient: monmap: 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:54.122 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.108+0000 7f893e6b8640 1 Processor -- start 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.108+0000 7f893e6b8640 1 -- start start 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.108+0000 7f893e6b8640 10 monclient: init 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f893e6b8640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:54.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f893e6b8640 10 monclient: _add_conns ranks=[4,3,5,8,1,7,0,6,2] 2024-09-17T15:51:54.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f893e6b8640 10 monclient(hunting): picked mon.noname-c con 0x7f89380a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:51:54.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f893e6b8640 10 monclient(hunting): picked mon.noname-g con 0x7f8938135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:54.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f893e6b8640 10 monclient(hunting): picked mon.noname-h con 0x7f8938135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:54.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f893e6b8640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8938134f20 con 0x7f8938135b50 2024-09-17T15:51:54.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f893e6b8640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89380a8e50 con 0x7f89380a5520 2024-09-17T15:51:54.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f893e6b8640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8938138800 con 0x7f8938135f70 2024-09-17T15:51:54.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f893e6b8640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:54.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f893e6b8640 10 monclient(hunting): authenticate will time out at 1111.784264s 2024-09-17T15:51:54.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8937fff640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f89380a5520 0x7f8938136d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:46818/0 (socket says 172.21.15.64:46818) 2024-09-17T15:51:54.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8937fff640 1 -- 172.21.15.64:0/3292095516 learned_addr learned my addr 172.21.15.64:0/3292095516 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2440617051 0 0) 0x7f89380a8e50 con 0x7f89380a5520 2024-09-17T15:51:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): my global_id is 154111 2024-09-17T15:51:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8924003520 con 0x7f89380a5520 2024-09-17T15:51:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3501137384 0 0) 0x7f8924003520 con 0x7f89380a5520 2024-09-17T15:51:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89380a8e50 con 0x7f89380a5520 2024-09-17T15:51:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8928001bb0 con 0x7f89380a5520 2024-09-17T15:51:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): dump: 2024-09-17T15:51:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.126 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2684709632 0 0) 0x7f8938134f20 con 0x7f8938135b50 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): my global_id is 115428 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8924003520 con 0x7f8938135b50 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2974805949 0 0) 0x7f8938138800 con 0x7f8938135f70 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 10 monclient(hunting): my global_id is 154121 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8938134f20 con 0x7f8938135f70 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.109+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3964424046 0 0) 0x7f89380a8e50 con 0x7f89380a5520 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 >> v1:172.21.15.100:6791/0 conn(0x7f8938135f70 legacy=0x7f8938137b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 >> v1:172.21.15.100:6790/0 conn(0x7f8938135b50 legacy=0x7f8938137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 10 monclient: _finish_hunting 0 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 1 monclient: found mon.c 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893813ab60 con 0x7f89380a5520 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 10 monclient: _finish_auth 0 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f893e6b8640 5 monclient: authenticate success, global_id 154111 2024-09-17T15:51:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f893e6b8640 10 monclient: _renew_subs 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f893e6b8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f893e6b8640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f89381421b0 con 0x7f89380a5520 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f893e6b8640 10 monclient: _renew_subs 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f893e6b8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f893e6b8640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f89381d7160 con 0x7f89380a5520 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89280038c0 con 0x7f89380a5520 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8928003e30 con 0x7f89380a5520 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f89367fc640 4 set_mon_vals no callback set 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f8934ff9640 10 monclient: dump: 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.129 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.130 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f893e6b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.110+0000 7f893e6b8640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f89180065c0 con 0x7f8918002860 2024-09-17T15:51:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f8928004050 con 0x7f89380a5520 2024-09-17T15:51:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(284..284 src has 192..284) ==== 3898+0+0 (unknown 1175997477 0 0) 0x7f892804ed60 con 0x7f89380a5520 2024-09-17T15:51:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1692439757 0 0) 0x7f89180065c0 con 0x7f8918002860 2024-09-17T15:51:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 10 monclient(hunting): my global_id is 124695 2024-09-17T15:51:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89380a8e50 con 0x7f8918002860 2024-09-17T15:51:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f893cc55640 10 monclient: get_auth_request con 0x7f8924040540 auth_method 0 2024-09-17T15:51:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2935489115 0 0) 0x7f89380a8e50 con 0x7f8918002860 2024-09-17T15:51:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89180065c0 con 0x7f8918002860 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f892c0019f0 con 0x7f8918002860 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 10 monclient: dump: 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.132 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.133 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.133 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.133 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.133 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.133 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1040443961 0 0) 0x7f89180065c0 con 0x7f8918002860 2024-09-17T15:51:54.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.111+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.64:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f891800a160 con 0x7f8918002860 2024-09-17T15:51:54.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.112+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f892c001ca0 con 0x7f8918002860 2024-09-17T15:51:54.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.112+0000 7f8934ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:54.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.112+0000 7f8934ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:54.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.112+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 >> v1:172.21.15.64:6789/0 conn(0x7f8918002860 legacy=0x7f8918004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.112+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 >> v1:172.21.15.64:6789/0 conn(0x7f8918002860 legacy=0x7f8918004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:54.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.118+0000 7f893e6b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:54.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.118+0000 7f893e6b8640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8918009e50 con 0x7f8918009110 2024-09-17T15:51:54.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.118+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1595720805 0 0) 0x7f8918009e50 con 0x7f8918009110 2024-09-17T15:51:54.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.118+0000 7f8934ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.118+0000 7f8934ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.118+0000 7f8934ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.118+0000 7f8934ff9640 10 monclient(hunting): my global_id is 124704 2024-09-17T15:51:54.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.118+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89180065c0 con 0x7f8918009110 2024-09-17T15:51:54.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 187095500 0 0) 0x7f89180065c0 con 0x7f8918009110 2024-09-17T15:51:54.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8918009e50 con 0x7f8918009110 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f892c003b60 con 0x7f8918009110 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f8934ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f8934ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f8934ff9640 10 monclient: dump: 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.136 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.137 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.137 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.137 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.137 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.137 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2453342346 0 0) 0x7f8918009e50 con 0x7f8918009110 2024-09-17T15:51:54.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 --> v1:172.21.15.64:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f891800c3f0 con 0x7f8918009110 2024-09-17T15:51:54.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1709751117) 0x7f892c0050d0 con 0x7f8918009110 2024-09-17T15:51:54.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f8934ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:54.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f8934ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:54.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 >> v1:172.21.15.64:6789/0 conn(0x7f8918009110 legacy=0x7f89180095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f8934ff9640 1 -- 172.21.15.64:0/3292095516 >> v1:172.21.15.64:6789/0 conn(0x7f8918009110 legacy=0x7f89180095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:54.138 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:54.138 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:54.138 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:54.138 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:54.138 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:54.138 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 121037, 2024-09-17T15:51:54.138 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:54.138 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:54.138 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:54.138 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:54.138 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:54.138 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:54.138 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.140 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:54.141 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.142 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.143 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:54.145 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.147 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:54.147 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:54.147 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.147 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.147 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.147 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:54.147 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:54.147 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.147 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:54.148 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.149 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:54.150 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:54.151 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.151 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.151 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.151 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:54.151 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:54.151 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.151 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.151 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.151 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:54.151 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.154 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:54.154 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:54.154 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.154 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.154 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.154 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:54.154 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.154 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.154 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:54.154 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:54.155 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.155 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.155 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.155 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.155 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.155 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:54.155 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:54.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.155 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.155 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.155 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:54.155 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:51:54.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.157 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.157 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:54.157 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:54.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f893e6b8640 1 -- 172.21.15.64:0/3292095516 >> v1:172.21.15.100:6812/4210963484 conn(0x7f8924040540 legacy=0x7f8924042a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f893e6b8640 10 monclient: shutdown 2024-09-17T15:51:54.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f893e6b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:54.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f893e6b8640 1 -- 172.21.15.64:0/3292095516 >> v1:172.21.15.64:6791/0 conn(0x7f89380a5520 legacy=0x7f8938136d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f893cc55640 1 -- 172.21.15.64:0/3292095516 reap_dead start 2024-09-17T15:51:54.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f893cc55640 1 -- 172.21.15.64:0/3292095516 reap_dead start 2024-09-17T15:51:54.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f893e6b8640 1 -- 172.21.15.64:0/3292095516 shutdown_connections 2024-09-17T15:51:54.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f893e6b8640 1 -- 172.21.15.64:0/3292095516 >> 172.21.15.64:0/3292095516 conn(0x7f893806db30 msgr2=0x7f89380a8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:54.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.119+0000 7f893e6b8640 1 -- 172.21.15.64:0/3292095516 shutdown_connections 2024-09-17T15:51:54.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.120+0000 7f893e6b8640 1 -- 172.21.15.64:0/3292095516 wait complete. 2024-09-17T15:51:54.159 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:54.159 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-17T15:51:54.159 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:51:54.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.259+0000 7f6e64d32640 10 monclient: get_monmap_and_config 2024-09-17T15:51:54.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.259+0000 7f6e64d32640 10 monclient: build_initial_monmap 2024-09-17T15:51:54.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.259+0000 7f6e64d32640 10 monclient: monmap: 2024-09-17T15:51:54.262 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:54.262 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.262 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:54.260809+0000 2024-09-17T15:51:54.262 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:54.260809+0000 2024-09-17T15:51:54.262 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:54.262 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.262 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:54.262 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:54.262 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:54.262 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:54.262 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:54.262 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:54.263 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:54.263 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:54.263 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:54.263 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.260+0000 7f6e64d32640 1 Processor -- start 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.260+0000 7f6e64d32640 1 -- start start 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.260+0000 7f6e64d32640 10 monclient: init 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e64d32640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e64d32640 10 monclient: _add_conns ranks=[0,3,2,1,6,8,5,4,7] 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e64d32640 10 monclient(hunting): picked mon.noname-a con 0x7f6e600a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e64d32640 10 monclient(hunting): picked mon.noname-g con 0x7f6e60135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:54.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e64d32640 10 monclient(hunting): picked mon.noname-b con 0x7f6e60135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e64d32640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e6009d6f0 con 0x7f6e600a5520 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e64d32640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e6009d9d0 con 0x7f6e60135f70 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e64d32640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e6009dcb0 con 0x7f6e60135b50 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e64d32640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e64d32640 10 monclient(hunting): authenticate will time out at 1111.936424s 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5e59c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f6e600a5520 0x7f6e600a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:54306/0 (socket says 172.21.15.64:54306) 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5e59c640 1 -- 172.21.15.64:0/3571972176 learned_addr learned my addr 172.21.15.64:0/3571972176 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 1 -- 172.21.15.64:0/3571972176 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3154977139 0 0) 0x7f6e6009d6f0 con 0x7f6e600a5520 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 10 monclient(hunting): my global_id is 124713 2024-09-17T15:51:54.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 1 -- 172.21.15.64:0/3571972176 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e48003520 con 0x7f6e600a5520 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 1 -- 172.21.15.64:0/3571972176 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3363738229 0 0) 0x7f6e6009d9d0 con 0x7f6e60135f70 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 10 monclient(hunting): my global_id is 124769 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 1 -- 172.21.15.64:0/3571972176 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e6009d6f0 con 0x7f6e60135f70 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 1 -- 172.21.15.64:0/3571972176 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1051330571 0 0) 0x7f6e48003520 con 0x7f6e600a5520 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 1 -- 172.21.15.64:0/3571972176 >> v1:172.21.15.100:6790/0 conn(0x7f6e60135b50 legacy=0x7f6e6013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 1 -- 172.21.15.64:0/3571972176 >> v1:172.21.15.64:6790/0 conn(0x7f6e60135f70 legacy=0x7f6e601415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 10 monclient: _finish_hunting 0 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 1 monclient: found mon.noname-a 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.261+0000 7f6e5d59a640 1 -- 172.21.15.64:0/3571972176 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e6009ccd0 con 0x7f6e600a5520 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e5d59a640 10 monclient: _finish_auth 0 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e5d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e64d32640 5 monclient: authenticate success, global_id 124713 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e5d59a640 1 -- 172.21.15.64:0/3571972176 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6e50001830 con 0x7f6e600a5520 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e64d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e5d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:54.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e5d59a640 10 monclient: dump: 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e64d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e5d59a640 1 -- 172.21.15.64:0/3571972176 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6e500019d0 con 0x7f6e600a5520 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e5d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e5d59a640 1 -- 172.21.15.64:0/3571972176 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6e50004be0 con 0x7f6e600a5520 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e64d32640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e64d32640 4 set_mon_vals no callback set 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e5d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e5d59a640 10 monclient: dump: 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e64d32640 10 monclient: shutdown 2024-09-17T15:51:54.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e64d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:54.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e64d32640 1 -- 172.21.15.64:0/3571972176 >> v1:172.21.15.64:6789/0 conn(0x7f6e600a5520 legacy=0x7f6e600a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e64d32640 1 -- 172.21.15.64:0/3571972176 shutdown_connections 2024-09-17T15:51:54.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e64d32640 1 -- 172.21.15.64:0/3571972176 >> 172.21.15.64:0/3571972176 conn(0x7f6e6006db30 msgr2=0x7f6e600ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:54.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e64d32640 1 -- 172.21.15.64:0/3571972176 shutdown_connections 2024-09-17T15:51:54.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.262+0000 7f6e64d32640 1 -- 172.21.15.64:0/3571972176 wait complete. 2024-09-17T15:51:54.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.263+0000 7f6e64d32640 10 monclient: build_initial_monmap 2024-09-17T15:51:54.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.263+0000 7f6e64d32640 10 monclient: monmap: 2024-09-17T15:51:54.277 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:54.277 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:54.277 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:54.277 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:54.277 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.263+0000 7f6e64d32640 1 Processor -- start 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.263+0000 7f6e64d32640 1 -- start start 2024-09-17T15:51:54.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.263+0000 7f6e64d32640 10 monclient: init 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e64d32640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e64d32640 10 monclient: _add_conns ranks=[4,5,7,3,2,6,8,1,0] 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e64d32640 10 monclient(hunting): picked mon.noname-c con 0x7f6e600a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e64d32640 10 monclient(hunting): picked mon.noname-h con 0x7f6e60135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e64d32640 10 monclient(hunting): picked mon.noname-i con 0x7f6e60135f70 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e64d32640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e6009e9a0 con 0x7f6e600a5520 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e64d32640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e600a0060 con 0x7f6e60135b50 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e64d32640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e600a0340 con 0x7f6e60135f70 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e64d32640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e5e59c640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f6e600a5520 0x7f6e601ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:46828/0 (socket says 172.21.15.64:46828) 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e64d32640 10 monclient(hunting): authenticate will time out at 1111.939553s 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e5e59c640 1 -- 172.21.15.64:0/3449965529 learned_addr learned my addr 172.21.15.64:0/3449965529 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2325916575 0 0) 0x7f6e6009e9a0 con 0x7f6e600a5520 2024-09-17T15:51:54.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e4effd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e4effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e4effd640 10 monclient(hunting): my global_id is 154120 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e40003520 con 0x7f6e600a5520 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3286025600 0 0) 0x7f6e600a0340 con 0x7f6e60135f70 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e4effd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e4effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e4effd640 10 monclient(hunting): my global_id is 154114 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e6009e9a0 con 0x7f6e60135f70 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.264+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 699835140 0 0) 0x7f6e600a0060 con 0x7f6e60135b50 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient(hunting): my global_id is 154130 2024-09-17T15:51:54.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e600a0340 con 0x7f6e60135b50 2024-09-17T15:51:54.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2105513600 0 0) 0x7f6e40003520 con 0x7f6e600a5520 2024-09-17T15:51:54.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e600a0060 con 0x7f6e600a5520 2024-09-17T15:51:54.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6e500017f0 con 0x7f6e600a5520 2024-09-17T15:51:54.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:54.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:54.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient(hunting): dump: 2024-09-17T15:51:54.281 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.281 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.281 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.281 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.281 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2568131212 0 0) 0x7f6e6009e9a0 con 0x7f6e60135f70 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e40003520 con 0x7f6e60135f70 2024-09-17T15:51:54.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6e58001d70 con 0x7f6e60135f70 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient(hunting): dump: 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.283 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1149937107 0 0) 0x7f6e600a0060 con 0x7f6e600a5520 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 >> v1:172.21.15.100:6792/0 conn(0x7f6e60135f70 legacy=0x7f6e601d5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 >> v1:172.21.15.100:6791/0 conn(0x7f6e60135b50 legacy=0x7f6e601ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient: _finish_hunting 0 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 monclient: found mon.c 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e601cafc0 con 0x7f6e600a5520 2024-09-17T15:51:54.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient: _finish_auth 0 2024-09-17T15:51:54.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:54.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:54.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e64d32640 5 monclient: authenticate success, global_id 154120 2024-09-17T15:51:54.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e64d32640 10 monclient: _renew_subs 2024-09-17T15:51:54.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e64d32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:54.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e64d32640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6e6009f040 con 0x7f6e600a5520 2024-09-17T15:51:54.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e64d32640 10 monclient: _renew_subs 2024-09-17T15:51:54.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e64d32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:54.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e64d32640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f6e600a8f70 con 0x7f6e600a5520 2024-09-17T15:51:54.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6e50001990 con 0x7f6e600a5520 2024-09-17T15:51:54.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:54.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6e50004b30 con 0x7f6e600a5520 2024-09-17T15:51:54.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e5cd99640 4 set_mon_vals no callback set 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.265+0000 7f6e4effd640 10 monclient: dump: 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.286 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e64d32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e64d32640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6e440065c0 con 0x7f6e44002860 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f6e50011920 con 0x7f6e600a5520 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(284..284 src has 192..284) ==== 3898+0+0 (unknown 1175997477 0 0) 0x7f6e5004ebb0 con 0x7f6e600a5520 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 293266882 0 0) 0x7f6e440065c0 con 0x7f6e44002860 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 10 monclient(hunting): my global_id is 124778 2024-09-17T15:51:54.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e600a0060 con 0x7f6e44002860 2024-09-17T15:51:54.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e5ed9d640 10 monclient: get_auth_request con 0x7f6e40040880 auth_method 0 2024-09-17T15:51:54.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4130094101 0 0) 0x7f6e600a0060 con 0x7f6e44002860 2024-09-17T15:51:54.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e440065c0 con 0x7f6e44002860 2024-09-17T15:51:54.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6e54003930 con 0x7f6e44002860 2024-09-17T15:51:54.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:54.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.266+0000 7f6e4effd640 10 monclient: dump: 2024-09-17T15:51:54.288 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.288 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.288 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.288 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.288 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.288 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.267+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1506844646 0 0) 0x7f6e440065c0 con 0x7f6e44002860 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.267+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.64:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6e4400a160 con 0x7f6e44002860 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.267+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6e54005b10 con 0x7f6e44002860 2024-09-17T15:51:54.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.267+0000 7f6e4effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:54.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.267+0000 7f6e4effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:54.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.267+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 >> v1:172.21.15.64:6790/0 conn(0x7f6e44002860 legacy=0x7f6e44004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.267+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 >> v1:172.21.15.64:6790/0 conn(0x7f6e44002860 legacy=0x7f6e44004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:54.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e64d32640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:54.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e64d32640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6e44009e70 con 0x7f6e44009130 2024-09-17T15:51:54.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 157303039 0 0) 0x7f6e44009e70 con 0x7f6e44009130 2024-09-17T15:51:54.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e4effd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e4effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e4effd640 10 monclient(hunting): my global_id is 124787 2024-09-17T15:51:54.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e440065c0 con 0x7f6e44009130 2024-09-17T15:51:54.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1792154362 0 0) 0x7f6e440065c0 con 0x7f6e44009130 2024-09-17T15:51:54.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e44009e70 con 0x7f6e44009130 2024-09-17T15:51:54.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6e54004290 con 0x7f6e44009130 2024-09-17T15:51:54.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:54.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.273+0000 7f6e4effd640 10 monclient: dump: 2024-09-17T15:51:54.292 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.292 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.292 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.292 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.292 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.293 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.293 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.293 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.293 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.293 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.293 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.293 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.293 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.293 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.293 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.294 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1553815206 0 0) 0x7f6e44009e70 con 0x7f6e44009130 2024-09-17T15:51:54.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 --> v1:172.21.15.64:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6e4400c410 con 0x7f6e44009130 2024-09-17T15:51:54.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3603159438) 0x7f6e54004560 con 0x7f6e44009130 2024-09-17T15:51:54.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e4effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:54.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e4effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:54.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 >> v1:172.21.15.64:6790/0 conn(0x7f6e44009130 legacy=0x7f6e440095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e4effd640 1 -- 172.21.15.64:0/3449965529 >> v1:172.21.15.64:6790/0 conn(0x7f6e44009130 legacy=0x7f6e440095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:54.294 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:54.294 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 121192, 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:54.295 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:54.296 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.296 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:51:54.296 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:54.296 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:54.296 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:54.296 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.296 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.296 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.296 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.296 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.296 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.296 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.296 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.297 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.297 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.297 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.297 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.297 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:54.297 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:54.297 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.297 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.297 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.297 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.297 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.297 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.297 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:54.298 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:54.299 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:54.299 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:54.299 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:54.299 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:54.299 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:54.299 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:54.299 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:54.299 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:54.299 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:54.299 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.299 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.299 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.300 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:54.301 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:54.301 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.301 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.301 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.301 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:54.301 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.301 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.301 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:54.301 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:54.301 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.303 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:54.304 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:54.305 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.305 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.305 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.305 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:54.305 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.305 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.305 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:54.305 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:54.305 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.305 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.306 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.306 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:54.306 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:54.306 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.306 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.306 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.306 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:54.306 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.306 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.306 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.307 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:54.308 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:54.309 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.309 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.309 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.309 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:54.309 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.309 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.309 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:54.309 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:54.309 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.309 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:54.311 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.312 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.313 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.313 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:54.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.313 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.313 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:51:54.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.313 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.313 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.313 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:54.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.313 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.313 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:54.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.314 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.314 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.314 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:51:54.314 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.314 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.314 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:54.314 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:54.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e64d32640 1 -- 172.21.15.64:0/3449965529 >> v1:172.21.15.100:6812/4210963484 conn(0x7f6e40040880 legacy=0x7f6e40042d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e64d32640 10 monclient: shutdown 2024-09-17T15:51:54.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e64d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:54.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e64d32640 1 -- 172.21.15.64:0/3449965529 >> v1:172.21.15.64:6791/0 conn(0x7f6e600a5520 legacy=0x7f6e601ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e5ed9d640 1 -- 172.21.15.64:0/3449965529 reap_dead start 2024-09-17T15:51:54.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e5ed9d640 1 -- 172.21.15.64:0/3449965529 reap_dead start 2024-09-17T15:51:54.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e64d32640 1 -- 172.21.15.64:0/3449965529 shutdown_connections 2024-09-17T15:51:54.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e64d32640 1 -- 172.21.15.64:0/3449965529 >> 172.21.15.64:0/3449965529 conn(0x7f6e6006db30 msgr2=0x7f6e600a63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:54.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e64d32640 1 -- 172.21.15.64:0/3449965529 shutdown_connections 2024-09-17T15:51:54.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.274+0000 7f6e64d32640 1 -- 172.21.15.64:0/3449965529 wait complete. 2024-09-17T15:51:54.316 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:54.316 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-17T15:51:54.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:51:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.416+0000 7fa3378b6640 10 monclient: get_monmap_and_config 2024-09-17T15:51:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.416+0000 7fa3378b6640 10 monclient: build_initial_monmap 2024-09-17T15:51:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.416+0000 7fa3378b6640 10 monclient: monmap: 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:54.417679+0000 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:54.417679+0000 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:54.420 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:54.420 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.417+0000 7fa3378b6640 1 Processor -- start 2024-09-17T15:51:54.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.417+0000 7fa3378b6640 1 -- start start 2024-09-17T15:51:54.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.417+0000 7fa3378b6640 10 monclient: init 2024-09-17T15:51:54.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:54.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 10 monclient: _add_conns ranks=[4,3,6,2,5,7,0,1,8] 2024-09-17T15:51:54.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 10 monclient(hunting): picked mon.noname-c con 0x7fa3300a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:51:54.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 10 monclient(hunting): picked mon.noname-g con 0x7fa330135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:54.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 10 monclient(hunting): picked mon.noname-d con 0x7fa330135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:51:54.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa33009d6f0 con 0x7fa330135b50 2024-09-17T15:51:54.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa33009d9d0 con 0x7fa3300a5520 2024-09-17T15:51:54.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa33009dcb0 con 0x7fa330135f70 2024-09-17T15:51:54.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:54.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 10 monclient(hunting): authenticate will time out at 1112.093229s 2024-09-17T15:51:54.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa335652640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fa3300a5520 0x7fa3300a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:46842/0 (socket says 172.21.15.64:46842) 2024-09-17T15:51:54.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa335652640 1 -- 172.21.15.64:0/1307202779 learned_addr learned my addr 172.21.15.64:0/1307202779 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:54.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 1 -- 172.21.15.64:0/1307202779 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2591832670 0 0) 0x7fa33009d9d0 con 0x7fa3300a5520 2024-09-17T15:51:54.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient(hunting): my global_id is 154129 2024-09-17T15:51:54.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 1 -- 172.21.15.64:0/1307202779 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa31c003520 con 0x7fa3300a5520 2024-09-17T15:51:54.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 1 -- 172.21.15.64:0/1307202779 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 46921127 0 0) 0x7fa33009dcb0 con 0x7fa330135f70 2024-09-17T15:51:54.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient(hunting): my global_id is 124710 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 1 -- 172.21.15.64:0/1307202779 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa33009d9d0 con 0x7fa330135f70 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 1 -- 172.21.15.64:0/1307202779 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2921230583 0 0) 0x7fa31c003520 con 0x7fa3300a5520 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 1 -- 172.21.15.64:0/1307202779 >> v1:172.21.15.64:6792/0 conn(0x7fa330135f70 legacy=0x7fa3301415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 1 -- 172.21.15.64:0/1307202779 >> v1:172.21.15.100:6790/0 conn(0x7fa330135b50 legacy=0x7fa33013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 1 monclient: found mon.noname-c 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 1 -- 172.21.15.64:0/1307202779 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33009ccd0 con 0x7fa3300a5520 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient: _finish_auth 0 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient: _check_auth_tickets 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 1 -- 172.21.15.64:0/1307202779 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa32c001830 con 0x7fa3300a5520 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 5 monclient: authenticate success, global_id 154129 2024-09-17T15:51:54.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:54.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:54.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient: dump: 2024-09-17T15:51:54.431 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.431 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.431 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.431 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.431 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.431 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.431 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.431 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.431 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.431 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 1 -- 172.21.15.64:0/1307202779 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa32c0019d0 con 0x7fa3300a5520 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 1 -- 172.21.15.64:0/1307202779 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa32c004be0 con 0x7fa3300a5520 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa3378b6640 4 set_mon_vals no callback set 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.418+0000 7fa327fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.419+0000 7fa327fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:54.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.419+0000 7fa327fff640 10 monclient: dump: 2024-09-17T15:51:54.433 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.433 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.433 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.433 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.433 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.433 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.433 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.433 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.433 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.433 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.433 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.433 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.434 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.434 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.434 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.434 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.419+0000 7fa3378b6640 10 monclient: shutdown 2024-09-17T15:51:54.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.419+0000 7fa3378b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:54.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.419+0000 7fa3378b6640 1 -- 172.21.15.64:0/1307202779 >> v1:172.21.15.64:6791/0 conn(0x7fa3300a5520 legacy=0x7fa3300a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.419+0000 7fa3378b6640 1 -- 172.21.15.64:0/1307202779 shutdown_connections 2024-09-17T15:51:54.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.419+0000 7fa3378b6640 1 -- 172.21.15.64:0/1307202779 >> 172.21.15.64:0/1307202779 conn(0x7fa33006db30 msgr2=0x7fa3300ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:54.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.419+0000 7fa3378b6640 1 -- 172.21.15.64:0/1307202779 shutdown_connections 2024-09-17T15:51:54.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.419+0000 7fa3378b6640 1 -- 172.21.15.64:0/1307202779 wait complete. 2024-09-17T15:51:54.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 10 monclient: build_initial_monmap 2024-09-17T15:51:54.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 10 monclient: monmap: 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:54.435 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:54.436 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:54.436 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 1 Processor -- start 2024-09-17T15:51:54.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 1 -- start start 2024-09-17T15:51:54.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 10 monclient: init 2024-09-17T15:51:54.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:54.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 10 monclient: _add_conns ranks=[4,3,2,8,0,1,6,5,7] 2024-09-17T15:51:54.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 10 monclient(hunting): picked mon.noname-c con 0x7fa3300a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:51:54.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 10 monclient(hunting): picked mon.noname-g con 0x7fa330135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:54.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 10 monclient(hunting): picked mon.noname-b con 0x7fa330135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:54.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa330133020 con 0x7fa330135f70 2024-09-17T15:51:54.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3300a04c0 con 0x7fa330135b50 2024-09-17T15:51:54.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa33009fc20 con 0x7fa3300a5520 2024-09-17T15:51:54.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:54.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.420+0000 7fa3378b6640 10 monclient(hunting): authenticate will time out at 1112.095990s 2024-09-17T15:51:54.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa335652640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fa3300a5520 0x7fa3301cd520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:46844/0 (socket says 172.21.15.64:46844) 2024-09-17T15:51:54.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa335652640 1 -- 172.21.15.64:0/2577173474 learned_addr learned my addr 172.21.15.64:0/2577173474 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:54.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3875685113 0 0) 0x7fa33009fc20 con 0x7fa3300a5520 2024-09-17T15:51:54.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): my global_id is 154138 2024-09-17T15:51:54.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa314003550 con 0x7fa3300a5520 2024-09-17T15:51:54.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2935313553 0 0) 0x7fa330133020 con 0x7fa330135f70 2024-09-17T15:51:54.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): my global_id is 124796 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa33009fc20 con 0x7fa330135f70 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 18008906 0 0) 0x7fa314003550 con 0x7fa3300a5520 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa330133020 con 0x7fa3300a5520 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa32c0047e0 con 0x7fa3300a5520 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): dump: 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.438 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4087226120 0 0) 0x7fa33009fc20 con 0x7fa330135f70 2024-09-17T15:51:54.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa314003550 con 0x7fa330135f70 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 971201479 0 0) 0x7fa3300a04c0 con 0x7fa330135b50 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): my global_id is 115455 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa33009fc20 con 0x7fa330135b50 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa328001cf0 con 0x7fa330135f70 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient(hunting): dump: 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.440 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.441 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.441 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.441 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.441 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.441 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.441 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.441 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.441 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.441 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.441 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.441 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.441 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3070868881 0 0) 0x7fa330133020 con 0x7fa3300a5520 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 >> v1:172.21.15.100:6790/0 conn(0x7fa330135b50 legacy=0x7fa3300a4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 >> v1:172.21.15.64:6790/0 conn(0x7fa330135f70 legacy=0x7fa3301d58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 monclient: found mon.c 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33009ecc0 con 0x7fa3300a5520 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient: _finish_auth 0 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa3378b6640 5 monclient: authenticate success, global_id 154138 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa3378b6640 10 monclient: _renew_subs 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa3378b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:54.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa3378b6640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa3300a5110 con 0x7fa3300a5520 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa32c004c30 con 0x7fa3300a5520 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa32c0051e0 con 0x7fa3300a5520 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa325ffb640 10 monclient: dump: 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.443 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.422+0000 7fa3378b6640 10 monclient: _renew_subs 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.421+0000 7fa3277fe640 4 set_mon_vals no callback set 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.422+0000 7fa3378b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.422+0000 7fa3378b6640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fa3301d71c0 con 0x7fa3300a5520 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.422+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fa32c011b40 con 0x7fa3300a5520 2024-09-17T15:51:54.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.422+0000 7fa3378b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.422+0000 7fa3378b6640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa3301351d0 con 0x7fa330142f10 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(284..284 src has 192..284) ==== 3898+0+0 (unknown 1175997477 0 0) 0x7fa32c04ec10 con 0x7fa3300a5520 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 601084314 0 0) 0x7fa33009fc20 con 0x7fa330142f10 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 10 monclient(hunting): my global_id is 154147 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa330133020 con 0x7fa330142f10 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa335e53640 10 monclient: get_auth_request con 0x7fa314040b70 auth_method 0 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4193934843 0 0) 0x7fa330133020 con 0x7fa330142f10 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa33009fc20 con 0x7fa330142f10 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa320001960 con 0x7fa330142f10 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 10 monclient: dump: 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.446 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.447 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.447 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.447 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3832903630 0 0) 0x7fa33009fc20 con 0x7fa330142f10 2024-09-17T15:51:54.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.423+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa330134dd0 con 0x7fa330142f10 2024-09-17T15:51:54.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.424+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa320001d30 con 0x7fa330142f10 2024-09-17T15:51:54.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.424+0000 7fa325ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:54.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.424+0000 7fa325ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:54.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.424+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 >> v1:172.21.15.64:6791/0 conn(0x7fa330142f10 legacy=0x7fa33012d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.424+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 >> v1:172.21.15.64:6791/0 conn(0x7fa330142f10 legacy=0x7fa33012d550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:54.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa3378b6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:54.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa3378b6640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa3301ce3e0 con 0x7fa33002e450 2024-09-17T15:51:54.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3104511765 0 0) 0x7fa320003610 con 0x7fa33002e450 2024-09-17T15:51:54.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 10 monclient(hunting): my global_id is 154156 2024-09-17T15:51:54.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa33009fc20 con 0x7fa33002e450 2024-09-17T15:51:54.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3224499438 0 0) 0x7fa33009fc20 con 0x7fa33002e450 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa320003610 con 0x7fa33002e450 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa320003d10 con 0x7fa33002e450 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 10 monclient: dump: 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2561070887 0 0) 0x7fa320003610 con 0x7fa33002e450 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.430+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 --> v1:172.21.15.64:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa330068b50 con 0x7fa33002e450 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 1497665281) 0x7fa3200051c0 con 0x7fa33002e450 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa325ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa325ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 >> v1:172.21.15.64:6791/0 conn(0x7fa33002e450 legacy=0x7fa33012dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa325ffb640 1 -- 172.21.15.64:0/2577173474 >> v1:172.21.15.64:6791/0 conn(0x7fa33002e450 legacy=0x7fa33012dc60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:54.452 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:54.452 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:54.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:54.452 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:54.452 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:54.452 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 5763, 2024-09-17T15:51:54.452 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:54.452 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:54.452 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:54.452 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:54.452 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.453 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.454 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:54.455 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:54.456 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:54.456 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:54.456 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:54.456 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:54.456 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:54.456 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:54.456 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:54.456 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:54.456 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:54.456 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:54.456 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:54.456 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.456 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:54.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:54.458 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.461 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.461 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:54.461 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:54.461 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.461 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.461 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:54.461 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:54.461 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.461 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.463 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:54.465 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.469 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.470 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.470 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:51:54.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.470 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.470 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:54.470 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:54.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa3378b6640 1 -- 172.21.15.64:0/2577173474 >> v1:172.21.15.100:6812/4210963484 conn(0x7fa314040b70 legacy=0x7fa314043030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa3378b6640 10 monclient: shutdown 2024-09-17T15:51:54.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa3378b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:54.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa3378b6640 1 -- 172.21.15.64:0/2577173474 >> v1:172.21.15.64:6791/0 conn(0x7fa3300a5520 legacy=0x7fa3301cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa335e53640 1 -- 172.21.15.64:0/2577173474 reap_dead start 2024-09-17T15:51:54.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa335e53640 1 -- 172.21.15.64:0/2577173474 reap_dead start 2024-09-17T15:51:54.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa3378b6640 1 -- 172.21.15.64:0/2577173474 shutdown_connections 2024-09-17T15:51:54.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa3378b6640 1 -- 172.21.15.64:0/2577173474 >> 172.21.15.64:0/2577173474 conn(0x7fa33006db30 msgr2=0x7fa3301334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:54.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa3378b6640 1 -- 172.21.15.64:0/2577173474 shutdown_connections 2024-09-17T15:51:54.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.431+0000 7fa3378b6640 1 -- 172.21.15.64:0/2577173474 wait complete. 2024-09-17T15:51:54.472 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:54.472 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-17T15:51:54.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-17T15:51:54.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.573+0000 7fefc0933640 10 monclient: get_monmap_and_config 2024-09-17T15:51:54.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.573+0000 7fefc0933640 10 monclient: build_initial_monmap 2024-09-17T15:51:54.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.573+0000 7fefc0933640 10 monclient: monmap: 2024-09-17T15:51:54.575 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:54.575 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.575 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:54.574384+0000 2024-09-17T15:51:54.575 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:54.574384+0000 2024-09-17T15:51:54.575 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:54.575 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.576 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:54.576 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:54.576 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:54.576 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:54.576 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:54.576 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:54.576 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:54.576 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:54.576 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:54.576 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefc0933640 1 Processor -- start 2024-09-17T15:51:54.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefc0933640 1 -- start start 2024-09-17T15:51:54.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefc0933640 10 monclient: init 2024-09-17T15:51:54.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefc0933640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:54.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefc0933640 10 monclient: _add_conns ranks=[2,7,6,8,0,5,1,4,3] 2024-09-17T15:51:54.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefc0933640 10 monclient(hunting): picked mon.noname-b con 0x7fefbc0a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:54.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefc0933640 10 monclient(hunting): picked mon.noname-i con 0x7fefbc135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:54.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefc0933640 10 monclient(hunting): picked mon.noname-d con 0x7fefbc135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:51:54.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefc0933640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefbc09d6f0 con 0x7fefbc0a5520 2024-09-17T15:51:54.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefc0933640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefbc09d9d0 con 0x7fefbc135f70 2024-09-17T15:51:54.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefc0933640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefbc09dcb0 con 0x7fefbc135b50 2024-09-17T15:51:54.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefc0933640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:54.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefc0933640 10 monclient(hunting): authenticate will time out at 1112.249909s 2024-09-17T15:51:54.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefbad9d640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fefbc135f70 0x7fefbc1415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:51426/0 (socket says 172.21.15.64:51426) 2024-09-17T15:51:54.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefbad9d640 1 -- 172.21.15.64:0/1481797962 learned_addr learned my addr 172.21.15.64:0/1481797962 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:54.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.574+0000 7fefba59c640 1 --1- 172.21.15.64:0/1481797962 >> v1:172.21.15.64:6790/0 conn(0x7fefbc0a5520 0x7fefbc0a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58418/0 (socket says 172.21.15.64:58418) 2024-09-17T15:51:54.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 -- 172.21.15.64:0/1481797962 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1842326178 0 0) 0x7fefbc09d9d0 con 0x7fefbc135f70 2024-09-17T15:51:54.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient(hunting): my global_id is 124719 2024-09-17T15:51:54.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 -- 172.21.15.64:0/1481797962 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefa4003520 con 0x7fefbc135f70 2024-09-17T15:51:54.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 -- 172.21.15.64:0/1481797962 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3610016519 0 0) 0x7fefbc09d6f0 con 0x7fefbc0a5520 2024-09-17T15:51:54.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient(hunting): my global_id is 124805 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 -- 172.21.15.64:0/1481797962 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefbc09d9d0 con 0x7fefbc0a5520 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 -- 172.21.15.64:0/1481797962 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 892191379 0 0) 0x7fefbc09dcb0 con 0x7fefbc135b50 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient(hunting): my global_id is 154123 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 -- 172.21.15.64:0/1481797962 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefbc09d6f0 con 0x7fefbc135b50 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 -- 172.21.15.64:0/1481797962 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 975538464 0 0) 0x7fefa4003520 con 0x7fefbc135f70 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 -- 172.21.15.64:0/1481797962 >> v1:172.21.15.100:6792/0 conn(0x7fefbc135b50 legacy=0x7fefbc13deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 -- 172.21.15.64:0/1481797962 >> v1:172.21.15.64:6790/0 conn(0x7fefbc0a5520 legacy=0x7fefbc0a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient: _finish_hunting 0 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 monclient: found mon.noname-d 2024-09-17T15:51:54.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 -- 172.21.15.64:0/1481797962 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefbc09ccd0 con 0x7fefbc135f70 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient: _finish_auth 0 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient: _check_auth_tickets 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 -- 172.21.15.64:0/1481797962 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fefb4001bb0 con 0x7fefbc135f70 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefc0933640 5 monclient: authenticate success, global_id 124719 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefc0933640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient: dump: 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.587 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefc0933640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 -- 172.21.15.64:0/1481797962 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fefb4001d50 con 0x7fefbc135f70 2024-09-17T15:51:54.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:54.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefc0933640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:54.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 1 -- 172.21.15.64:0/1481797962 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fefb4004f60 con 0x7fefbc135f70 2024-09-17T15:51:54.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefc0933640 4 set_mon_vals no callback set 2024-09-17T15:51:54.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:51:54.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.575+0000 7fefb959a640 10 monclient: dump: 2024-09-17T15:51:54.589 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.589 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.589 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.589 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.589 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.589 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.589 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.576+0000 7fefc0933640 10 monclient: shutdown 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.576+0000 7fefc0933640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.576+0000 7fefc0933640 1 -- 172.21.15.64:0/1481797962 >> v1:172.21.15.64:6792/0 conn(0x7fefbc135f70 legacy=0x7fefbc1415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.576+0000 7fefc0933640 1 -- 172.21.15.64:0/1481797962 shutdown_connections 2024-09-17T15:51:54.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.576+0000 7fefc0933640 1 -- 172.21.15.64:0/1481797962 >> 172.21.15.64:0/1481797962 conn(0x7fefbc06db30 msgr2=0x7fefbc0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:54.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.576+0000 7fefc0933640 1 -- 172.21.15.64:0/1481797962 shutdown_connections 2024-09-17T15:51:54.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.576+0000 7fefc0933640 1 -- 172.21.15.64:0/1481797962 wait complete. 2024-09-17T15:51:54.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.576+0000 7fefc0933640 10 monclient: build_initial_monmap 2024-09-17T15:51:54.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.576+0000 7fefc0933640 10 monclient: monmap: 2024-09-17T15:51:54.591 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:54.591 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:54.591 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:54.591 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:54.591 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:54.591 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:54.591 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:54.591 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:54.591 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:54.592 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:54.592 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:54.592 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:54.592 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:54.592 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:54.592 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:54.592 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefc0933640 1 Processor -- start 2024-09-17T15:51:54.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefc0933640 1 -- start start 2024-09-17T15:51:54.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefc0933640 10 monclient: init 2024-09-17T15:51:54.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefc0933640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:54.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefc0933640 10 monclient: _add_conns ranks=[7,3,2,1,6,4,5,0,8] 2024-09-17T15:51:54.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefc0933640 10 monclient(hunting): picked mon.noname-i con 0x7fefbc0a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:54.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefc0933640 10 monclient(hunting): picked mon.noname-g con 0x7fefbc135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:54.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefc0933640 10 monclient(hunting): picked mon.noname-b con 0x7fefbc135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:54.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefc0933640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefbc135270 con 0x7fefbc135f70 2024-09-17T15:51:54.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefc0933640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefbc0a6760 con 0x7fefbc135b50 2024-09-17T15:51:54.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefc0933640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefbc138380 con 0x7fefbc0a5520 2024-09-17T15:51:54.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefc0933640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:54.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefc0933640 10 monclient(hunting): authenticate will time out at 1112.252764s 2024-09-17T15:51:54.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefbad9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fefbc135f70 0x7fefbc1d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58434/0 (socket says 172.21.15.64:58434) 2024-09-17T15:51:54.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.577+0000 7fefbad9d640 1 -- 172.21.15.64:0/341409492 learned_addr learned my addr 172.21.15.64:0/341409492 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:54.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1333795022 0 0) 0x7fefbc135270 con 0x7fefbc135f70 2024-09-17T15:51:54.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): my global_id is 124814 2024-09-17T15:51:54.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef9c003520 con 0x7fefbc135f70 2024-09-17T15:51:54.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2994841596 0 0) 0x7fef9c003520 con 0x7fefbc135f70 2024-09-17T15:51:54.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefbc135270 con 0x7fefbc135f70 2024-09-17T15:51:54.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fefb4001b70 con 0x7fefbc135f70 2024-09-17T15:51:54.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:54.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:51:54.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): dump: 2024-09-17T15:51:54.594 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.595 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4224334498 0 0) 0x7fefbc138380 con 0x7fefbc0a5520 2024-09-17T15:51:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): my global_id is 154132 2024-09-17T15:51:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef9c003520 con 0x7fefbc0a5520 2024-09-17T15:51:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 633930341 0 0) 0x7fefbc0a6760 con 0x7fefbc135b50 2024-09-17T15:51:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient(hunting): my global_id is 115464 2024-09-17T15:51:54.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefbc138380 con 0x7fefbc135b50 2024-09-17T15:51:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2787817242 0 0) 0x7fefbc135270 con 0x7fefbc135f70 2024-09-17T15:51:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 >> v1:172.21.15.100:6792/0 conn(0x7fefbc0a5520 legacy=0x7fefbc136fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 >> v1:172.21.15.100:6790/0 conn(0x7fefbc135b50 legacy=0x7fefbc1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient: _finish_hunting 0 2024-09-17T15:51:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 monclient: found mon.b 2024-09-17T15:51:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:51:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefbc1389e0 con 0x7fefbc135f70 2024-09-17T15:51:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient: _finish_auth 0 2024-09-17T15:51:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefc0933640 5 monclient: authenticate success, global_id 124814 2024-09-17T15:51:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefc0933640 10 monclient: _renew_subs 2024-09-17T15:51:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefc0933640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:51:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefc0933640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fefbc13ad40 con 0x7fefbc135f70 2024-09-17T15:51:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefc0933640 10 monclient: _renew_subs 2024-09-17T15:51:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefc0933640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:51:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefc0933640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fefbc137fd0 con 0x7fefbc135f70 2024-09-17T15:51:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fefb4004e10 con 0x7fefbc135f70 2024-09-17T15:51:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fefb40053c0 con 0x7fefbc135f70 2024-09-17T15:51:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefb8d99640 4 set_mon_vals no callback set 2024-09-17T15:51:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.578+0000 7fefaaffd640 10 monclient: dump: 2024-09-17T15:51:54.598 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.600 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.579+0000 7fefc0933640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.579+0000 7fefc0933640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fefa00065c0 con 0x7fefa0002860 2024-09-17T15:51:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.579+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fefb40055e0 con 0x7fefbc135f70 2024-09-17T15:51:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.579+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(284..284 src has 192..284) ==== 3898+0+0 (unknown 1175997477 0 0) 0x7fefb404ef80 con 0x7fefbc135f70 2024-09-17T15:51:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3697251703 0 0) 0x7fefa00065c0 con 0x7fefa0002860 2024-09-17T15:51:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 10 monclient(hunting): my global_id is 124728 2024-09-17T15:51:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefbc135270 con 0x7fefa0002860 2024-09-17T15:51:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefb9d9b640 10 monclient: get_auth_request con 0x7fef9c040510 auth_method 0 2024-09-17T15:51:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 552840233 0 0) 0x7fefbc135270 con 0x7fefa0002860 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefa00065c0 con 0x7fefa0002860 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fefac001430 con 0x7fefa0002860 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 10 monclient: dump: 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.601 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.602 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.602 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.602 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.602 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.602 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.602 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.602 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 440292023 0 0) 0x7fefa00065c0 con 0x7fefa0002860 2024-09-17T15:51:54.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.64:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fefa000a160 con 0x7fefa0002860 2024-09-17T15:51:54.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fefac0017a0 con 0x7fefa0002860 2024-09-17T15:51:54.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:54.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:54.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 >> v1:172.21.15.64:6792/0 conn(0x7fefa0002860 legacy=0x7fefa0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.580+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 >> v1:172.21.15.64:6792/0 conn(0x7fefa0002860 legacy=0x7fefa0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:54.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.586+0000 7fefc0933640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:54.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.586+0000 7fefc0933640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fefa0009e70 con 0x7fefa0009130 2024-09-17T15:51:54.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2158992463 0 0) 0x7fefa0009e70 con 0x7fefa0009130 2024-09-17T15:51:54.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 10 monclient(hunting): my global_id is 124737 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefa00065c0 con 0x7fefa0009130 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1313788670 0 0) 0x7fefa00065c0 con 0x7fefa0009130 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefa0009e70 con 0x7fefa0009130 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fefac004dd0 con 0x7fefa0009130 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 10 monclient: dump: 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.605 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3036046471 0 0) 0x7fefa0009e70 con 0x7fefa0009130 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 --> v1:172.21.15.64:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fefa000c410 con 0x7fefa0009130 2024-09-17T15:51:54.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1685711494) 0x7fefac0062b0 con 0x7fefa0009130 2024-09-17T15:51:54.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:54.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:54.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 >> v1:172.21.15.64:6792/0 conn(0x7fefa0009130 legacy=0x7fefa00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.587+0000 7fefaaffd640 1 -- 172.21.15.64:0/341409492 >> v1:172.21.15.64:6792/0 conn(0x7fefa0009130 legacy=0x7fefa00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:54.607 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:54.607 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:54.607 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:54.607 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:54.607 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:54.607 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 121505, 2024-09-17T15:51:54.608 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:54.608 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:54.608 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:54.608 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:54.608 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:54.608 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:54.608 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:54.608 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:54.608 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:54.608 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:54.608 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.608 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:51:54.608 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.609 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.611 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:54.611 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:54.611 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:54.611 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:54.611 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:54.611 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:54.611 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:54.611 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:54.611 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:54.611 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:54.611 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:54.611 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:54.611 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.612 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:54.613 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.614 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.614 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:54.614 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:54.614 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.614 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.614 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.614 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:54.614 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:54.614 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.617 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:54.618 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.620 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:54.620 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:54.620 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.620 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.620 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.620 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:54.620 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.620 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.620 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.622 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.623 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.624 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.625 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.626 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.626 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:54.626 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.626 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.626 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:54.626 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:54.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.588+0000 7fefc0933640 1 -- 172.21.15.64:0/341409492 >> v1:172.21.15.100:6812/4210963484 conn(0x7fef9c040510 legacy=0x7fef9c0429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.588+0000 7fefc0933640 10 monclient: shutdown 2024-09-17T15:51:54.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.588+0000 7fefc0933640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:54.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.588+0000 7fefc0933640 1 -- 172.21.15.64:0/341409492 >> v1:172.21.15.64:6790/0 conn(0x7fefbc135f70 legacy=0x7fefbc1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.588+0000 7fefbad9d640 1 -- 172.21.15.64:0/341409492 reap_dead start 2024-09-17T15:51:54.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.588+0000 7fefbad9d640 1 -- 172.21.15.64:0/341409492 reap_dead start 2024-09-17T15:51:54.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.588+0000 7fefc0933640 1 -- 172.21.15.64:0/341409492 shutdown_connections 2024-09-17T15:51:54.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.588+0000 7fefc0933640 1 -- 172.21.15.64:0/341409492 >> 172.21.15.64:0/341409492 conn(0x7fefbc06db30 msgr2=0x7fefbc133f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:54.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.588+0000 7fefc0933640 1 -- 172.21.15.64:0/341409492 shutdown_connections 2024-09-17T15:51:54.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.588+0000 7fefc0933640 1 -- 172.21.15.64:0/341409492 wait complete. 2024-09-17T15:51:54.628 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:54.628 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-17T15:51:54.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-17T15:51:54.710 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39586]: starting. 2024-09-17T15:51:54.710 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39586]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39584 2024-09-17T15:51:54.710 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39586]: shutting down. 2024-09-17T15:51:54.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.737+0000 7f0d3f744640 10 monclient: get_monmap_and_config 2024-09-17T15:51:54.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.737+0000 7f0d3f744640 10 monclient: build_initial_monmap 2024-09-17T15:51:54.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.737+0000 7f0d3f744640 10 monclient: monmap: 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:54.737949+0000 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:54.737949+0000 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:54.739 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:54.740 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:54.740 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.737+0000 7f0d3f744640 1 Processor -- start 2024-09-17T15:51:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.737+0000 7f0d3f744640 1 -- start start 2024-09-17T15:51:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.737+0000 7f0d3f744640 10 monclient: init 2024-09-17T15:51:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d3f744640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d3f744640 10 monclient: _add_conns ranks=[1,7,8,0,3,5,6,2,4] 2024-09-17T15:51:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d3f744640 10 monclient(hunting): picked mon.noname-f con 0x7f0d380a54b0 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d3f744640 10 monclient(hunting): picked mon.noname-i con 0x7f0d38135c80 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d3f744640 10 monclient(hunting): picked mon.noname-e con 0x7f0d381360a0 addr v1:172.21.15.64:6793/0 2024-09-17T15:51:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d3f744640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d380aacb0 con 0x7f0d380a54b0 2024-09-17T15:51:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d3f744640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d3809d780 con 0x7f0d38135c80 2024-09-17T15:51:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d3f744640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d3809da60 con 0x7f0d381360a0 2024-09-17T15:51:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d3f744640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d3f744640 10 monclient(hunting): authenticate will time out at 1112.413415s 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d3dce1640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f0d381360a0 0x7f0d38141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:37918/0 (socket says 172.21.15.64:37918) 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d3dce1640 1 -- 172.21.15.64:0/4204124401 learned_addr learned my addr 172.21.15.64:0/4204124401 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 1 -- 172.21.15.64:0/4204124401 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3727585757 0 0) 0x7f0d3809da60 con 0x7f0d381360a0 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 10 monclient(hunting): my global_id is 154124 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 1 -- 172.21.15.64:0/4204124401 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d20003520 con 0x7f0d381360a0 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 1 -- 172.21.15.64:0/4204124401 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1838779543 0 0) 0x7f0d3809d780 con 0x7f0d38135c80 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 10 monclient(hunting): my global_id is 154177 2024-09-17T15:51:54.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 1 -- 172.21.15.64:0/4204124401 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d3809da60 con 0x7f0d38135c80 2024-09-17T15:51:54.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 1 -- 172.21.15.64:0/4204124401 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4293681601 0 0) 0x7f0d380aacb0 con 0x7f0d380a54b0 2024-09-17T15:51:54.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 10 monclient(hunting): my global_id is 6535 2024-09-17T15:51:54.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 1 -- 172.21.15.64:0/4204124401 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d3809d780 con 0x7f0d380a54b0 2024-09-17T15:51:54.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 1 -- 172.21.15.64:0/4204124401 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 403983792 0 0) 0x7f0d20003520 con 0x7f0d381360a0 2024-09-17T15:51:54.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 1 -- 172.21.15.64:0/4204124401 >> v1:172.21.15.100:6792/0 conn(0x7f0d38135c80 legacy=0x7f0d3813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.738+0000 7f0d37fff640 1 -- 172.21.15.64:0/4204124401 >> v1:172.21.15.100:6789/0 conn(0x7f0d380a54b0 legacy=0x7f0d380a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:54.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 1 monclient: found mon.noname-e 2024-09-17T15:51:54.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:54.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 1 -- 172.21.15.64:0/4204124401 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3809dd40 con 0x7f0d381360a0 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 10 monclient: _finish_auth 0 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 10 monclient: _check_auth_tickets 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d3f744640 5 monclient: authenticate success, global_id 154124 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 1 -- 172.21.15.64:0/4204124401 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0d28001830 con 0x7f0d381360a0 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d3f744640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 10 monclient: dump: 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.751 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.751 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.751 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.751 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.751 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.751 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.751 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.751 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.751 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.751 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.751 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.751 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d3f744640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 1 -- 172.21.15.64:0/4204124401 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0d280019d0 con 0x7f0d381360a0 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d3f744640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 1 -- 172.21.15.64:0/4204124401 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0d28004be0 con 0x7f0d381360a0 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d3f744640 4 set_mon_vals no callback set 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d37fff640 10 monclient: dump: 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.752 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d3f744640 10 monclient: shutdown 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d3f744640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d3f744640 1 -- 172.21.15.64:0/4204124401 >> v1:172.21.15.64:6793/0 conn(0x7f0d381360a0 legacy=0x7f0d38141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d3f744640 1 -- 172.21.15.64:0/4204124401 shutdown_connections 2024-09-17T15:51:54.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d3f744640 1 -- 172.21.15.64:0/4204124401 >> 172.21.15.64:0/4204124401 conn(0x7f0d3806daa0 msgr2=0x7f0d380ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d3f744640 1 -- 172.21.15.64:0/4204124401 shutdown_connections 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.739+0000 7f0d3f744640 1 -- 172.21.15.64:0/4204124401 wait complete. 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.740+0000 7f0d3f744640 10 monclient: build_initial_monmap 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.740+0000 7f0d3f744640 10 monclient: monmap: 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:54.754 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.740+0000 7f0d3f744640 1 Processor -- start 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.740+0000 7f0d3f744640 1 -- start start 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.740+0000 7f0d3f744640 10 monclient: init 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d3f744640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d3f744640 10 monclient: _add_conns ranks=[6,8,4,0,2,3,5,1,7] 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d3f744640 10 monclient(hunting): picked mon.noname-d con 0x7f0d380a54b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d3f744640 10 monclient(hunting): picked mon.noname-e con 0x7f0d38135c80 addr v1:172.21.15.64:6793/0 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d3f744640 10 monclient(hunting): picked mon.noname-c con 0x7f0d381360a0 addr v1:172.21.15.64:6791/0 2024-09-17T15:51:54.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d3f744640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d381cebf0 con 0x7f0d381360a0 2024-09-17T15:51:54.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d3f744640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d380a7000 con 0x7f0d380a54b0 2024-09-17T15:51:54.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d3f744640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d380a0710 con 0x7f0d38135c80 2024-09-17T15:51:54.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d3f744640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:54.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d3f744640 10 monclient(hunting): authenticate will time out at 1112.416308s 2024-09-17T15:51:54.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d3d4e0640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f0d380a54b0 0x7f0d381cddd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:51458/0 (socket says 172.21.15.64:51458) 2024-09-17T15:51:54.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d3ccdf640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f0d38135c80 0x7f0d381ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:37920/0 (socket says 172.21.15.64:37920) 2024-09-17T15:51:54.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d3d4e0640 1 -- 172.21.15.64:0/2692831812 learned_addr learned my addr 172.21.15.64:0/2692831812 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:54.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1491045896 0 0) 0x7f0d380a7000 con 0x7f0d380a54b0 2024-09-17T15:51:54.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): my global_id is 124746 2024-09-17T15:51:54.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d24003520 con 0x7f0d380a54b0 2024-09-17T15:51:54.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2199069050 0 0) 0x7f0d380a0710 con 0x7f0d38135c80 2024-09-17T15:51:54.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): my global_id is 154133 2024-09-17T15:51:54.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d380a7000 con 0x7f0d38135c80 2024-09-17T15:51:54.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2974668489 0 0) 0x7f0d381cebf0 con 0x7f0d381360a0 2024-09-17T15:51:54.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): my global_id is 154183 2024-09-17T15:51:54.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d380a0710 con 0x7f0d381360a0 2024-09-17T15:51:54.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4041415789 0 0) 0x7f0d380a7000 con 0x7f0d38135c80 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d381cebf0 con 0x7f0d38135c80 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0d2c001a50 con 0x7f0d38135c80 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): dump: 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.758 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 661856154 0 0) 0x7f0d24003520 con 0x7f0d380a54b0 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d380a7000 con 0x7f0d380a54b0 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0d30001eb0 con 0x7f0d380a54b0 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 10 monclient(hunting): dump: 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.759 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.760 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.760 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.760 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.760 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.760 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.760 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.760 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.760 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.760 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.760 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.760 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.760 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.760 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.741+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3065673842 0 0) 0x7f0d380a0710 con 0x7f0d381360a0 2024-09-17T15:51:54.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d24003520 con 0x7f0d381360a0 2024-09-17T15:51:54.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0d280016f0 con 0x7f0d381360a0 2024-09-17T15:51:54.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:54.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:54.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 10 monclient(hunting): dump: 2024-09-17T15:51:54.761 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.761 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.761 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.761 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.761 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.761 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.761 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3297140515 0 0) 0x7f0d381cebf0 con 0x7f0d38135c80 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 >> v1:172.21.15.64:6792/0 conn(0x7f0d380a54b0 legacy=0x7f0d381cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 >> v1:172.21.15.64:6791/0 conn(0x7f0d381360a0 legacy=0x7f0d381d6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:51:54.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 1 monclient: found mon.e 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d380a09f0 con 0x7f0d38135c80 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 10 monclient: _finish_auth 0 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d3f744640 5 monclient: authenticate success, global_id 154133 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d3f744640 10 monclient: _renew_subs 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d3f744640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d3f744640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0d380a51a0 con 0x7f0d38135c80 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0d2c0038c0 con 0x7f0d38135c80 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0d2c003eb0 con 0x7f0d38135c80 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:51:54.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d35ffb640 10 monclient: dump: 2024-09-17T15:51:54.764 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.764 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.764 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.764 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.764 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.764 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.764 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.764 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.764 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.764 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.764 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.764 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.764 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.765 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d377fe640 4 set_mon_vals no callback set 2024-09-17T15:51:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d3f744640 10 monclient: _renew_subs 2024-09-17T15:51:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d3f744640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.742+0000 7f0d3f744640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f0d38132520 con 0x7f0d38135c80 2024-09-17T15:51:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d3f744640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d3f744640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0d38077070 con 0x7f0d3812d810 2024-09-17T15:51:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f0d2c0040d0 con 0x7f0d38135c80 2024-09-17T15:51:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(285..285 src has 192..285) ==== 3506+0+0 (unknown 2220307653 0 0) 0x7f0d2c04e9d0 con 0x7f0d38135c80 2024-09-17T15:51:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1784709658 0 0) 0x7f0d3809d780 con 0x7f0d3812d810 2024-09-17T15:51:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d35ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d35ffb640 10 monclient(hunting): my global_id is 154142 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d381cebf0 con 0x7f0d3812d810 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d3dce1640 10 monclient: get_auth_request con 0x7f0d24040d40 auth_method 0 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2532189566 0 0) 0x7f0d381cebf0 con 0x7f0d3812d810 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d3809d780 con 0x7f0d3812d810 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0d30004080 con 0x7f0d3812d810 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.743+0000 7f0d35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.744+0000 7f0d35ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.744+0000 7f0d35ffb640 10 monclient: dump: 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.766 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.767 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.767 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.767 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.767 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.767 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.767 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.767 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.767 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.767 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.767 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.767 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.767 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.767 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.744+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4152944923 0 0) 0x7f0d3809d780 con 0x7f0d3812d810 2024-09-17T15:51:54.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.744+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d380ab5a0 con 0x7f0d3812d810 2024-09-17T15:51:54.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.744+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0d30005b40 con 0x7f0d3812d810 2024-09-17T15:51:54.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.744+0000 7f0d35ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:54.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.744+0000 7f0d35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:54.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.744+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 >> v1:172.21.15.64:6793/0 conn(0x7f0d3812d810 legacy=0x7f0d3812dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.744+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 >> v1:172.21.15.64:6793/0 conn(0x7f0d3812d810 legacy=0x7f0d3812dbf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:54.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.750+0000 7f0d3f744640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:54.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.750+0000 7f0d3f744640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0d380ac2a0 con 0x7f0d3812e300 2024-09-17T15:51:54.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.750+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3623658212 0 0) 0x7f0d300043a0 con 0x7f0d3812e300 2024-09-17T15:51:54.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.750+0000 7f0d35ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.750+0000 7f0d35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.750+0000 7f0d35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.750+0000 7f0d35ffb640 10 monclient(hunting): my global_id is 154151 2024-09-17T15:51:54.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.750+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d3809d780 con 0x7f0d3812e300 2024-09-17T15:51:54.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.751+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3755552672 0 0) 0x7f0d3809d780 con 0x7f0d3812e300 2024-09-17T15:51:54.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.751+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d300043a0 con 0x7f0d3812e300 2024-09-17T15:51:54.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.751+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0d30006030 con 0x7f0d3812e300 2024-09-17T15:51:54.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.751+0000 7f0d35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.751+0000 7f0d35ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:51:54.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.751+0000 7f0d35ffb640 10 monclient: dump: 2024-09-17T15:51:54.771 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.771 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.771 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.771 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.771 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.771 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.772 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.772 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.772 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.772 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.772 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.772 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.772 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.772 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.772 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.772 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.751+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3285048365 0 0) 0x7f0d300043a0 con 0x7f0d3812e300 2024-09-17T15:51:54.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.751+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 --> v1:172.21.15.64:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0d380a7330 con 0x7f0d3812e300 2024-09-17T15:51:54.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.751+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 4139194781) 0x7f0d30008210 con 0x7f0d3812e300 2024-09-17T15:51:54.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.751+0000 7f0d35ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:54.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.751+0000 7f0d35ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:54.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.751+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 >> v1:172.21.15.64:6793/0 conn(0x7f0d3812e300 legacy=0x7f0d381db770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.751+0000 7f0d35ffb640 1 -- 172.21.15.64:0/2692831812 >> v1:172.21.15.64:6793/0 conn(0x7f0d3812e300 legacy=0x7f0d381db770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:54.773 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:54.773 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:54.773 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:54.773 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:54.773 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:54.773 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 6087, 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:54.774 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:54.775 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:54.775 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.775 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.775 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.775 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.775 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.775 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.775 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.775 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.775 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.775 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.775 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.775 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.776 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:54.777 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.778 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.780 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:54.781 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.782 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.782 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.782 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:54.782 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.782 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.782 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.782 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:54.782 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:54.782 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.782 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.782 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:54.783 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.786 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.786 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:54.786 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.786 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.786 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.786 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:54.786 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:54.786 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.786 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.786 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.787 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.789 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:54.789 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:54.789 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.789 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.789 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.789 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:54.789 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:54.789 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.789 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.789 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:54.790 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.791 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:54.792 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:54.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.752+0000 7f0d3f744640 1 -- 172.21.15.64:0/2692831812 >> v1:172.21.15.100:6812/4210963484 conn(0x7f0d24040d40 legacy=0x7f0d24043200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.752+0000 7f0d3f744640 10 monclient: shutdown 2024-09-17T15:51:54.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.752+0000 7f0d3f744640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:54.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.752+0000 7f0d3f744640 1 -- 172.21.15.64:0/2692831812 >> v1:172.21.15.64:6793/0 conn(0x7f0d38135c80 legacy=0x7f0d381ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.752+0000 7f0d3dce1640 1 -- 172.21.15.64:0/2692831812 reap_dead start 2024-09-17T15:51:54.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.752+0000 7f0d3dce1640 1 -- 172.21.15.64:0/2692831812 reap_dead start 2024-09-17T15:51:54.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.752+0000 7f0d3f744640 1 -- 172.21.15.64:0/2692831812 shutdown_connections 2024-09-17T15:51:54.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.752+0000 7f0d3f744640 1 -- 172.21.15.64:0/2692831812 >> 172.21.15.64:0/2692831812 conn(0x7f0d3806daa0 msgr2=0x7f0d38133930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:54.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.752+0000 7f0d3f744640 1 -- 172.21.15.64:0/2692831812 shutdown_connections 2024-09-17T15:51:54.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.752+0000 7f0d3f744640 1 -- 172.21.15.64:0/2692831812 wait complete. 2024-09-17T15:51:54.794 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:54.794 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-17T15:51:54.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-17T15:51:54.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.895+0000 7f4954532640 10 monclient: get_monmap_and_config 2024-09-17T15:51:54.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.895+0000 7f4954532640 10 monclient: build_initial_monmap 2024-09-17T15:51:54.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.895+0000 7f4954532640 10 monclient: monmap: 2024-09-17T15:51:54.897 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:54.897 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.897 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:54.895976+0000 2024-09-17T15:51:54.897 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:54.895976+0000 2024-09-17T15:51:54.897 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:54.897 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.897 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:54.897 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:54.897 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:54.897 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:54.898 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:54.898 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:54.898 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:54.898 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:54.898 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:54.898 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.895+0000 7f4954532640 1 Processor -- start 2024-09-17T15:51:54.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.895+0000 7f4954532640 1 -- start start 2024-09-17T15:51:54.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.895+0000 7f4954532640 10 monclient: init 2024-09-17T15:51:54.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f4954532640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:54.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f4954532640 10 monclient: _add_conns ranks=[4,3,5,8,6,7,2,0,1] 2024-09-17T15:51:54.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f4954532640 10 monclient(hunting): picked mon.noname-c con 0x7f494c0a54b0 addr v1:172.21.15.64:6791/0 2024-09-17T15:51:54.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f4954532640 10 monclient(hunting): picked mon.noname-g con 0x7f494c135c80 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:54.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f4954532640 10 monclient(hunting): picked mon.noname-h con 0x7f494c1360a0 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f4954532640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f494c0aacb0 con 0x7f494c135c80 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f4954532640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f494c09d780 con 0x7f494c0a54b0 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f4954532640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f494c09da60 con 0x7f494c1360a0 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f4954532640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f4954532640 10 monclient(hunting): authenticate will time out at 1112.571462s 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f49522ce640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f494c0a54b0 0x7f494c0a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:46880/0 (socket says 172.21.15.64:46880) 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f49522ce640 1 -- 172.21.15.64:0/3642550449 learned_addr learned my addr 172.21.15.64:0/3642550449 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f49512cc640 1 -- 172.21.15.64:0/3642550449 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3075638568 0 0) 0x7f494c09d780 con 0x7f494c0a54b0 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f49512cc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f49512cc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f49512cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f49512cc640 10 monclient(hunting): my global_id is 154192 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.896+0000 7f49512cc640 1 -- 172.21.15.64:0/3642550449 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4938003520 con 0x7f494c0a54b0 2024-09-17T15:51:54.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 1 -- 172.21.15.64:0/3642550449 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1219417725 0 0) 0x7f4938003520 con 0x7f494c0a54b0 2024-09-17T15:51:54.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 1 -- 172.21.15.64:0/3642550449 >> v1:172.21.15.100:6791/0 conn(0x7f494c1360a0 legacy=0x7f494c141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 1 -- 172.21.15.64:0/3642550449 >> v1:172.21.15.100:6790/0 conn(0x7f494c135c80 legacy=0x7f494c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 10 monclient: _finish_hunting 0 2024-09-17T15:51:54.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 1 monclient: found mon.noname-c 2024-09-17T15:51:54.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:54.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:51:54.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 1 -- 172.21.15.64:0/3642550449 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f494c09dd40 con 0x7f494c0a54b0 2024-09-17T15:51:54.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 10 monclient: _finish_auth 0 2024-09-17T15:51:54.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 10 monclient: _check_auth_tickets 2024-09-17T15:51:54.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:54.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 1 -- 172.21.15.64:0/3642550449 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4948001830 con 0x7f494c0a54b0 2024-09-17T15:51:54.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 10 monclient: dump: 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.909 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f4954532640 5 monclient: authenticate success, global_id 154192 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f4954532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 1 -- 172.21.15.64:0/3642550449 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f49480019d0 con 0x7f494c0a54b0 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 1 -- 172.21.15.64:0/3642550449 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4948004be0 con 0x7f494c0a54b0 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f49512cc640 10 monclient: dump: 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.910 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.911 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.911 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.911 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.911 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.911 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.911 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.911 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.911 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.911 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.911 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.911 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.911 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.911 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f4954532640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:54.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f4954532640 4 set_mon_vals no callback set 2024-09-17T15:51:54.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f4954532640 10 monclient: shutdown 2024-09-17T15:51:54.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f4954532640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:54.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f4954532640 1 -- 172.21.15.64:0/3642550449 >> v1:172.21.15.64:6791/0 conn(0x7f494c0a54b0 legacy=0x7f494c0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f4954532640 1 -- 172.21.15.64:0/3642550449 shutdown_connections 2024-09-17T15:51:54.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f4954532640 1 -- 172.21.15.64:0/3642550449 >> 172.21.15.64:0/3642550449 conn(0x7f494c06daa0 msgr2=0x7f494c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:54.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f4954532640 1 -- 172.21.15.64:0/3642550449 shutdown_connections 2024-09-17T15:51:54.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.897+0000 7f4954532640 1 -- 172.21.15.64:0/3642550449 wait complete. 2024-09-17T15:51:54.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.898+0000 7f4954532640 10 monclient: build_initial_monmap 2024-09-17T15:51:54.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.898+0000 7f4954532640 10 monclient: monmap: 2024-09-17T15:51:54.912 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:54.912 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:54.913 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.898+0000 7f4954532640 1 Processor -- start 2024-09-17T15:51:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.898+0000 7f4954532640 1 -- start start 2024-09-17T15:51:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.898+0000 7f4954532640 10 monclient: init 2024-09-17T15:51:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4954532640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4954532640 10 monclient: _add_conns ranks=[0,1,3,7,5,8,2,4,6] 2024-09-17T15:51:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4954532640 10 monclient(hunting): picked mon.noname-a con 0x7f494c0a54b0 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4954532640 10 monclient(hunting): picked mon.noname-f con 0x7f494c135c80 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4954532640 10 monclient(hunting): picked mon.noname-g con 0x7f494c1360a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4954532640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f494c13abe0 con 0x7f494c0a54b0 2024-09-17T15:51:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4954532640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f494c0a6c20 con 0x7f494c135c80 2024-09-17T15:51:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4954532640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f494c138880 con 0x7f494c1360a0 2024-09-17T15:51:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4954532640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:54.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4954532640 10 monclient(hunting): authenticate will time out at 1112.574192s 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f49522ce640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f494c0a54b0 0x7f494c136de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:54314/0 (socket says 172.21.15.64:54314) 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f49522ce640 1 -- 172.21.15.64:0/679999428 learned_addr learned my addr 172.21.15.64:0/679999428 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3953433204 0 0) 0x7f494c13abe0 con 0x7f494c0a54b0 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): my global_id is 124749 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4930003520 con 0x7f494c0a54b0 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 105779769 0 0) 0x7f494c138880 con 0x7f494c1360a0 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): my global_id is 115491 2024-09-17T15:51:54.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f494c13abe0 con 0x7f494c1360a0 2024-09-17T15:51:54.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3105819873 0 0) 0x7f494c0a6c20 con 0x7f494c135c80 2024-09-17T15:51:54.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): my global_id is 6544 2024-09-17T15:51:54.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f494c138880 con 0x7f494c135c80 2024-09-17T15:51:54.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4115030356 0 0) 0x7f4930003520 con 0x7f494c0a54b0 2024-09-17T15:51:54.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f494c0a6c20 con 0x7f494c0a54b0 2024-09-17T15:51:54.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f49480017f0 con 0x7f494c0a54b0 2024-09-17T15:51:54.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:54.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:54.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): dump: 2024-09-17T15:51:54.916 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.917 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2972984226 0 0) 0x7f494c13abe0 con 0x7f494c1360a0 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4930003520 con 0x7f494c1360a0 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3189326626 0 0) 0x7f494c138880 con 0x7f494c135c80 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f494c13abe0 con 0x7f494c135c80 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4944001cb0 con 0x7f494c1360a0 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): dump: 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.918 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f493c0019d0 con 0x7f494c135c80 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:54.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 10 monclient(hunting): dump: 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.920 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.921 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.921 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.921 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3181384194 0 0) 0x7f494c0a6c20 con 0x7f494c0a54b0 2024-09-17T15:51:54.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.899+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 >> v1:172.21.15.100:6790/0 conn(0x7f494c1360a0 legacy=0x7f494c1d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 >> v1:172.21.15.100:6789/0 conn(0x7f494c135c80 legacy=0x7f494c1374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:51:54.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 1 monclient: found mon.a 2024-09-17T15:51:54.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:54.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:51:54.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f494c137c00 con 0x7f494c0a54b0 2024-09-17T15:51:54.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 10 monclient: _finish_auth 0 2024-09-17T15:51:54.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:54.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:54.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4954532640 5 monclient: authenticate success, global_id 124749 2024-09-17T15:51:54.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4954532640 10 monclient: _renew_subs 2024-09-17T15:51:54.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4954532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:51:54.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4954532640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f494c13a900 con 0x7f494c0a54b0 2024-09-17T15:51:54.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4954532640 10 monclient: _renew_subs 2024-09-17T15:51:54.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4954532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:51:54.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4954532640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f494c142430 con 0x7f494c0a54b0 2024-09-17T15:51:54.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4948001990 con 0x7f494c0a54b0 2024-09-17T15:51:54.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:54.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4948004bb0 con 0x7f494c0a54b0 2024-09-17T15:51:54.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4950acb640 4 set_mon_vals no callback set 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 10 monclient: dump: 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.923 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4954532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4954532640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f49340065c0 con 0x7f4934002860 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.900+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f4948011960 con 0x7f494c0a54b0 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(285..285 src has 192..285) ==== 3506+0+0 (unknown 2220307653 0 0) 0x7f494804ea90 con 0x7f494c0a54b0 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2511308627 0 0) 0x7f49340065c0 con 0x7f4934002860 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 10 monclient(hunting): my global_id is 6553 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f494c0a6c20 con 0x7f4934002860 2024-09-17T15:51:54.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4952acf640 10 monclient: get_auth_request con 0x7f4930040e00 auth_method 0 2024-09-17T15:51:54.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1423960300 0 0) 0x7f494c0a6c20 con 0x7f4934002860 2024-09-17T15:51:54.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49340065c0 con 0x7f4934002860 2024-09-17T15:51:54.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f493c003930 con 0x7f4934002860 2024-09-17T15:51:54.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:51:54.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 10 monclient: dump: 2024-09-17T15:51:54.925 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.925 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.925 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.925 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.925 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.925 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.925 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2397269333 0 0) 0x7f49340065c0 con 0x7f4934002860 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.901+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.100:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f493400a160 con 0x7f4934002860 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.902+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f493c003b10 con 0x7f4934002860 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.902+0000 7f4942ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.902+0000 7f4942ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:54.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.902+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 >> v1:172.21.15.100:6789/0 conn(0x7f4934002860 legacy=0x7f4934004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.902+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 >> v1:172.21.15.100:6789/0 conn(0x7f4934002860 legacy=0x7f4934004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:54.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.908+0000 7f4954532640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:54.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.908+0000 7f4954532640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4934009e20 con 0x7f49340090e0 2024-09-17T15:51:54.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.908+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3944563766 0 0) 0x7f4934009e20 con 0x7f49340090e0 2024-09-17T15:51:54.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.908+0000 7f4942ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:54.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.908+0000 7f4942ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:54.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.908+0000 7f4942ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:54.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.908+0000 7f4942ffd640 10 monclient(hunting): my global_id is 6562 2024-09-17T15:51:54.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.908+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49340065c0 con 0x7f49340090e0 2024-09-17T15:51:54.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.909+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1835406997 0 0) 0x7f49340065c0 con 0x7f49340090e0 2024-09-17T15:51:54.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.909+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4934009e20 con 0x7f49340090e0 2024-09-17T15:51:54.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.909+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f493c004300 con 0x7f49340090e0 2024-09-17T15:51:54.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.909+0000 7f4942ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:54.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.909+0000 7f4942ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:51:54.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.909+0000 7f4942ffd640 10 monclient: dump: 2024-09-17T15:51:54.929 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:54.929 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:54.929 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.909+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 389342684 0 0) 0x7f4934009e20 con 0x7f49340090e0 2024-09-17T15:51:54.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.909+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 --> v1:172.21.15.100:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f493400c3f0 con 0x7f49340090e0 2024-09-17T15:51:54.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.909+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7880 (unknown 0 0 1770135816) 0x7f493c0065a0 con 0x7f49340090e0 2024-09-17T15:51:54.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.909+0000 7f4942ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:54.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.909+0000 7f4942ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:54.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.909+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 >> v1:172.21.15.100:6789/0 conn(0x7f49340090e0 legacy=0x7f4934009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.909+0000 7f4942ffd640 1 -- 172.21.15.64:0/679999428 >> v1:172.21.15.100:6789/0 conn(0x7f49340090e0 legacy=0x7f4934009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:54.931 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:54.931 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:54.931 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:54.931 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "leader", 2024-09-17T15:51:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 371559, 2024-09-17T15:51:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.933 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.934 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:54.935 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:54.936 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:54.937 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:54.937 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:54.937 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.937 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:54.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.937 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:54.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.937 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:54.937 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:54.937 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.937 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.937 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:54.938 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.939 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:54.943 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:54.944 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.944 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.944 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.944 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:54.944 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.944 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.944 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:54.944 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:54.944 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.944 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:54.946 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:54.947 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:54.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.948 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:54.948 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:54.948 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:54.948 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.948 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.948 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:54.948 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:54.948 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:54.948 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:54.948 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:54.948 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.949 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:51:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:51:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": [ 2024-09-17T15:51:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:54.950 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:54.951 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:54.951 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:54.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:54.951 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:54.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:54.951 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:54.951 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:54.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.910+0000 7f4954532640 1 -- 172.21.15.64:0/679999428 >> v1:172.21.15.100:6812/4210963484 conn(0x7f4930040e00 legacy=0x7f49300432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.910+0000 7f4954532640 10 monclient: shutdown 2024-09-17T15:51:54.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.910+0000 7f4954532640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:54.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.910+0000 7f4954532640 1 -- 172.21.15.64:0/679999428 >> v1:172.21.15.64:6789/0 conn(0x7f494c0a54b0 legacy=0x7f494c136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:54.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.910+0000 7f4952acf640 1 -- 172.21.15.64:0/679999428 reap_dead start 2024-09-17T15:51:54.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.910+0000 7f4952acf640 1 -- 172.21.15.64:0/679999428 reap_dead start 2024-09-17T15:51:54.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.910+0000 7f4954532640 1 -- 172.21.15.64:0/679999428 shutdown_connections 2024-09-17T15:51:54.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.910+0000 7f4954532640 1 -- 172.21.15.64:0/679999428 >> 172.21.15.64:0/679999428 conn(0x7f494c06daa0 msgr2=0x7f494c134980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:54.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.910+0000 7f4954532640 1 -- 172.21.15.64:0/679999428 shutdown_connections 2024-09-17T15:51:54.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:54.910+0000 7f4954532640 1 -- 172.21.15.64:0/679999428 wait complete. 2024-09-17T15:51:54.953 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:54.953 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-17T15:51:54.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-17T15:51:55.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.053+0000 7ffae732b640 10 monclient: get_monmap_and_config 2024-09-17T15:51:55.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.053+0000 7ffae732b640 10 monclient: build_initial_monmap 2024-09-17T15:51:55.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.053+0000 7ffae732b640 10 monclient: monmap: 2024-09-17T15:51:55.055 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:55.054690+0000 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:55.054690+0000 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:55.056 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:55.057 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.054+0000 7ffae732b640 1 Processor -- start 2024-09-17T15:51:55.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.054+0000 7ffae732b640 1 -- start start 2024-09-17T15:51:55.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.054+0000 7ffae732b640 10 monclient: init 2024-09-17T15:51:55.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae732b640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:55.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae732b640 10 monclient: _add_conns ranks=[0,7,2,5,1,8,4,3,6] 2024-09-17T15:51:55.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae732b640 10 monclient(hunting): picked mon.noname-a con 0x7ffae00a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:55.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae732b640 10 monclient(hunting): picked mon.noname-i con 0x7ffae0135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:55.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae732b640 10 monclient(hunting): picked mon.noname-b con 0x7ffae0135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:55.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae732b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae009d6f0 con 0x7ffae00a5520 2024-09-17T15:51:55.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae732b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae009d9d0 con 0x7ffae0135f70 2024-09-17T15:51:55.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae732b640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae009dcb0 con 0x7ffae0135b50 2024-09-17T15:51:55.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae732b640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:55.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae732b640 10 monclient(hunting): authenticate will time out at 1112.730189s 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae50c7640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7ffae00a5520 0x7ffae00a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:54326/0 (socket says 172.21.15.64:54326) 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae58c8640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7ffae0135f70 0x7ffae01415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58442/0 (socket says 172.21.15.64:58442) 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae50c7640 1 -- 172.21.15.64:0/435716247 learned_addr learned my addr 172.21.15.64:0/435716247 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 1 -- 172.21.15.64:0/435716247 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1094285763 0 0) 0x7ffae009d6f0 con 0x7ffae00a5520 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient(hunting): my global_id is 124758 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 1 -- 172.21.15.64:0/435716247 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffacc003520 con 0x7ffae00a5520 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 1 -- 172.21.15.64:0/435716247 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3766210941 0 0) 0x7ffae009d9d0 con 0x7ffae0135f70 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient(hunting): my global_id is 124832 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 1 -- 172.21.15.64:0/435716247 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffae009d6f0 con 0x7ffae0135f70 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 1 -- 172.21.15.64:0/435716247 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3875261822 0 0) 0x7ffae009dcb0 con 0x7ffae0135b50 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient(hunting): my global_id is 154186 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 1 -- 172.21.15.64:0/435716247 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffae009d9d0 con 0x7ffae0135b50 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 1 -- 172.21.15.64:0/435716247 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 924249982 0 0) 0x7ffae009d6f0 con 0x7ffae0135f70 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 1 -- 172.21.15.64:0/435716247 >> v1:172.21.15.100:6792/0 conn(0x7ffae0135b50 legacy=0x7ffae013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 1 -- 172.21.15.64:0/435716247 >> v1:172.21.15.64:6789/0 conn(0x7ffae00a5520 legacy=0x7ffae00a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 1 monclient: found mon.noname-b 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:51:55.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 1 -- 172.21.15.64:0/435716247 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae009ccd0 con 0x7ffae0135f70 2024-09-17T15:51:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient: _finish_auth 0 2024-09-17T15:51:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:51:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 1 -- 172.21.15.64:0/435716247 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffad8001b90 con 0x7ffae0135f70 2024-09-17T15:51:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae732b640 5 monclient: authenticate success, global_id 124832 2024-09-17T15:51:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae732b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:51:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffad7fff640 10 monclient: dump: 2024-09-17T15:51:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.068 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.055+0000 7ffae732b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffad7fff640 1 -- 172.21.15.64:0/435716247 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffad8001db0 con 0x7ffae0135f70 2024-09-17T15:51:55.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffad7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffad7fff640 1 -- 172.21.15.64:0/435716247 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffad8004ea0 con 0x7ffae0135f70 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffad7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffad7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffad7fff640 10 monclient: dump: 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.070 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffae732b640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffae732b640 4 set_mon_vals no callback set 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffae732b640 10 monclient: shutdown 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffae732b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffae732b640 1 -- 172.21.15.64:0/435716247 >> v1:172.21.15.64:6790/0 conn(0x7ffae0135f70 legacy=0x7ffae01415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffae732b640 1 -- 172.21.15.64:0/435716247 shutdown_connections 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffae732b640 1 -- 172.21.15.64:0/435716247 >> 172.21.15.64:0/435716247 conn(0x7ffae006db30 msgr2=0x7ffae00ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:55.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffae732b640 1 -- 172.21.15.64:0/435716247 shutdown_connections 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.056+0000 7ffae732b640 1 -- 172.21.15.64:0/435716247 wait complete. 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.057+0000 7ffae732b640 10 monclient: build_initial_monmap 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.057+0000 7ffae732b640 10 monclient: monmap: 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:55.072 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.057+0000 7ffae732b640 1 Processor -- start 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.057+0000 7ffae732b640 1 -- start start 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.057+0000 7ffae732b640 10 monclient: init 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffae732b640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffae732b640 10 monclient: _add_conns ranks=[1,2,6,5,3,4,0,7,8] 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffae732b640 10 monclient(hunting): picked mon.noname-f con 0x7ffae00a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffae732b640 10 monclient(hunting): picked mon.noname-b con 0x7ffae0135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffae732b640 10 monclient(hunting): picked mon.noname-d con 0x7ffae0135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:51:55.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffae732b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae0134f50 con 0x7ffae00a5520 2024-09-17T15:51:55.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffae732b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae00a0690 con 0x7ffae0135b50 2024-09-17T15:51:55.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffae732b640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae009f820 con 0x7ffae0135f70 2024-09-17T15:51:55.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffae732b640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:55.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffae732b640 10 monclient(hunting): authenticate will time out at 1112.733181s 2024-09-17T15:51:55.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffae48c6640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7ffae0135b50 0x7ffae01d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58450/0 (socket says 172.21.15.64:58450) 2024-09-17T15:51:55.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffae48c6640 1 -- 172.21.15.64:0/3530446922 learned_addr learned my addr 172.21.15.64:0/3530446922 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:55.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 911686330 0 0) 0x7ffae009f820 con 0x7ffae0135f70 2024-09-17T15:51:55.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): my global_id is 124755 2024-09-17T15:51:55.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffac4003550 con 0x7ffae0135f70 2024-09-17T15:51:55.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 159355289 0 0) 0x7ffae00a0690 con 0x7ffae0135b50 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): my global_id is 124841 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffae009f820 con 0x7ffae0135b50 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1694675320 0 0) 0x7ffadc001000 con 0x7ffae00a5520 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): my global_id is 6571 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffae00a0690 con 0x7ffae00a5520 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 241591081 0 0) 0x7ffac4003550 con 0x7ffae0135f70 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffadc001000 con 0x7ffae0135f70 2024-09-17T15:51:55.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffad8001f60 con 0x7ffae0135f70 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): dump: 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.076 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4033477357 0 0) 0x7ffae009f820 con 0x7ffae0135b50 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffac4003550 con 0x7ffae0135b50 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffad0001b90 con 0x7ffae0135b50 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient(hunting): dump: 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.077 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3323399246 0 0) 0x7ffadc001000 con 0x7ffae0135f70 2024-09-17T15:51:55.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 >> v1:172.21.15.64:6790/0 conn(0x7ffae0135b50 legacy=0x7ffae01d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 >> v1:172.21.15.100:6789/0 conn(0x7ffae00a5520 legacy=0x7ffae00a4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:51:55.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 monclient: found mon.d 2024-09-17T15:51:55.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:55.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:51:55.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae009fdf0 con 0x7ffae0135f70 2024-09-17T15:51:55.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:51:55.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:51:55.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.058+0000 7ffad5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:55.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffae732b640 5 monclient: authenticate success, global_id 124755 2024-09-17T15:51:55.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffae732b640 10 monclient: _renew_subs 2024-09-17T15:51:55.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffae732b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffae732b640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffae00a5200 con 0x7ffae0135f70 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffad8001b70 con 0x7ffae0135f70 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffad5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffad8005490 con 0x7ffae0135f70 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffad5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffad5ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffad5ffb640 10 monclient: dump: 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.080 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffad77fe640 4 set_mon_vals no callback set 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffae732b640 10 monclient: _renew_subs 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffae732b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffae732b640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7ffae009f070 con 0x7ffae0135f70 2024-09-17T15:51:55.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7ffad8011d70 con 0x7ffae0135f70 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffae732b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.059+0000 7ffae732b640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ffae00008d0 con 0x7ffae0142f10 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(285..285 src has 192..285) ==== 3506+0+0 (unknown 2220307653 0 0) 0x7ffad804ed00 con 0x7ffae0135f70 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffae48c6640 10 monclient: get_auth_request con 0x7ffac40408a0 auth_method 0 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3817233444 0 0) 0x7ffae00008d0 con 0x7ffae0142f10 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 10 monclient(hunting): my global_id is 115500 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffadc001000 con 0x7ffae0142f10 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4242629990 0 0) 0x7ffadc001000 con 0x7ffae0142f10 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffae00008d0 con 0x7ffae0142f10 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffadc004d10 con 0x7ffae0142f10 2024-09-17T15:51:55.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 10 monclient: dump: 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.083 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.084 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.084 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.084 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 601089492 0 0) 0x7ffae00008d0 con 0x7ffae0142f10 2024-09-17T15:51:55.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.060+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.100:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffae00a0030 con 0x7ffae0142f10 2024-09-17T15:51:55.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.061+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ffadc006210 con 0x7ffae0142f10 2024-09-17T15:51:55.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.061+0000 7ffad5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:55.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.061+0000 7ffad5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:55.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.061+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 >> v1:172.21.15.100:6790/0 conn(0x7ffae0142f10 legacy=0x7ffae012d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.061+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 >> v1:172.21.15.100:6790/0 conn(0x7ffae0142f10 legacy=0x7ffae012d660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:55.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.067+0000 7ffae732b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:55.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.067+0000 7ffae732b640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ffae01ce0a0 con 0x7ffae01cdcc0 2024-09-17T15:51:55.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.067+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3594123417 0 0) 0x7ffae01ce0a0 con 0x7ffae01cdcc0 2024-09-17T15:51:55.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.067+0000 7ffad5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.067+0000 7ffad5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.067+0000 7ffad5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.067+0000 7ffad5ffb640 10 monclient(hunting): my global_id is 115509 2024-09-17T15:51:55.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.067+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffae00008d0 con 0x7ffae01cdcc0 2024-09-17T15:51:55.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 53382849 0 0) 0x7ffae00008d0 con 0x7ffae01cdcc0 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffae01ce0a0 con 0x7ffae01cdcc0 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffadc006a30 con 0x7ffae01cdcc0 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffad5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffad5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffad5ffb640 10 monclient: dump: 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.087 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3234163951 0 0) 0x7ffae01ce0a0 con 0x7ffae01cdcc0 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 --> v1:172.21.15.100:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffae012e480 con 0x7ffae01cdcc0 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1432149737) 0x7ffadc006ae0 con 0x7ffae01cdcc0 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffad5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffad5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 >> v1:172.21.15.100:6790/0 conn(0x7ffae01cdcc0 legacy=0x7ffae012dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffad5ffb640 1 -- 172.21.15.64:0/3530446922 >> v1:172.21.15.100:6790/0 conn(0x7ffae01cdcc0 legacy=0x7ffae012dd70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:55.089 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:55.089 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:55.089 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:55.089 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:55.089 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:55.089 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 238290, 2024-09-17T15:51:55.089 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:55.089 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:55.089 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:55.089 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:55.090 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:55.091 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:55.091 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:55.091 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:55.091 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:55.091 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:55.091 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:55.091 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:55.091 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:55.091 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:55.091 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:55.091 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:55.091 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:55.091 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:55.092 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:55.093 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:55.093 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:55.093 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:55.093 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:55.093 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:55.093 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:55.093 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:55.093 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:55.093 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:55.093 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:55.093 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:55.093 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:55.093 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:55.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.095 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:55.096 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.097 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.097 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:55.097 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:55.097 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.097 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.097 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.097 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:55.097 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:55.097 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:55.099 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.100 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.101 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.101 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:55.101 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:55.101 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.101 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.101 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.101 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:55.101 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:55.101 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.101 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.103 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.105 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.106 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.107 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:55.108 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:55.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.108 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:55.108 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:55.108 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:51:55.108 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.108 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.108 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:55.108 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:55.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffae732b640 1 -- 172.21.15.64:0/3530446922 >> v1:172.21.15.100:6812/4210963484 conn(0x7ffac40408a0 legacy=0x7ffac4042d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffae732b640 10 monclient: shutdown 2024-09-17T15:51:55.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffae732b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:55.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffae732b640 1 -- 172.21.15.64:0/3530446922 >> v1:172.21.15.64:6792/0 conn(0x7ffae0135f70 legacy=0x7ffae01d62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffae58c8640 1 -- 172.21.15.64:0/3530446922 reap_dead start 2024-09-17T15:51:55.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.068+0000 7ffae58c8640 1 -- 172.21.15.64:0/3530446922 reap_dead start 2024-09-17T15:51:55.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.069+0000 7ffae732b640 1 -- 172.21.15.64:0/3530446922 shutdown_connections 2024-09-17T15:51:55.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.069+0000 7ffae732b640 1 -- 172.21.15.64:0/3530446922 >> 172.21.15.64:0/3530446922 conn(0x7ffae006db30 msgr2=0x7ffae012c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:55.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.069+0000 7ffae732b640 1 -- 172.21.15.64:0/3530446922 shutdown_connections 2024-09-17T15:51:55.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.069+0000 7ffae732b640 1 -- 172.21.15.64:0/3530446922 wait complete. 2024-09-17T15:51:55.110 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:55.110 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-17T15:51:55.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-17T15:51:55.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.212+0000 7fce3f4e2640 10 monclient: get_monmap_and_config 2024-09-17T15:51:55.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.212+0000 7fce3f4e2640 10 monclient: build_initial_monmap 2024-09-17T15:51:55.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.212+0000 7fce3f4e2640 10 monclient: monmap: 2024-09-17T15:51:55.214 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:55.214 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.214 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:55.212924+0000 2024-09-17T15:51:55.214 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:55.212924+0000 2024-09-17T15:51:55.214 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:55.214 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.214 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:55.214 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:55.215 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:55.215 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:55.215 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:55.215 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:55.215 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:55.215 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:55.215 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:55.215 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.212+0000 7fce3f4e2640 1 Processor -- start 2024-09-17T15:51:55.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.212+0000 7fce3f4e2640 1 -- start start 2024-09-17T15:51:55.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.212+0000 7fce3f4e2640 10 monclient: init 2024-09-17T15:51:55.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce3f4e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:55.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce3f4e2640 10 monclient: _add_conns ranks=[2,0,1,4,5,3,6,8,7] 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce3f4e2640 10 monclient(hunting): picked mon.noname-b con 0x7fce380a54b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce3f4e2640 10 monclient(hunting): picked mon.noname-a con 0x7fce38135c80 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce3f4e2640 10 monclient(hunting): picked mon.noname-f con 0x7fce381360a0 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce3f4e2640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce380aacb0 con 0x7fce38135c80 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce3f4e2640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce3809d780 con 0x7fce381360a0 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce3f4e2640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce3809da60 con 0x7fce380a54b0 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce3f4e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce3f4e2640 10 monclient(hunting): authenticate will time out at 1112.888494s 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce3ca7d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fce38135c80 0x7fce3813df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:42570/0 (socket says 172.21.15.64:42570) 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce3d27e640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fce380a54b0 0x7fce380a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51416/0 (socket says 172.21.15.64:51416) 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce3ca7d640 1 -- 172.21.15.64:0/1049940194 learned_addr learned my addr 172.21.15.64:0/1049940194 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce37fff640 1 -- 172.21.15.64:0/1049940194 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2198050435 0 0) 0x7fce380aacb0 con 0x7fce38135c80 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce37fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce37fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce37fff640 10 monclient(hunting): my global_id is 124767 2024-09-17T15:51:55.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce37fff640 1 -- 172.21.15.64:0/1049940194 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce20003520 con 0x7fce38135c80 2024-09-17T15:51:55.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce37fff640 1 -- 172.21.15.64:0/1049940194 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1398752509 0 0) 0x7fce3809da60 con 0x7fce380a54b0 2024-09-17T15:51:55.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce37fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce37fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce37fff640 10 monclient(hunting): my global_id is 124850 2024-09-17T15:51:55.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.213+0000 7fce37fff640 1 -- 172.21.15.64:0/1049940194 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce380aacb0 con 0x7fce380a54b0 2024-09-17T15:51:55.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 1 -- 172.21.15.64:0/1049940194 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 525096310 0 0) 0x7fce3809d780 con 0x7fce381360a0 2024-09-17T15:51:55.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient(hunting): my global_id is 6580 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 1 -- 172.21.15.64:0/1049940194 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce3809da60 con 0x7fce381360a0 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 1 -- 172.21.15.64:0/1049940194 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3893643503 0 0) 0x7fce20003520 con 0x7fce38135c80 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 1 -- 172.21.15.64:0/1049940194 >> v1:172.21.15.64:6790/0 conn(0x7fce380a54b0 legacy=0x7fce380a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 1 -- 172.21.15.64:0/1049940194 >> v1:172.21.15.100:6789/0 conn(0x7fce381360a0 legacy=0x7fce38141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 1 monclient: found mon.noname-a 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 1 -- 172.21.15.64:0/1049940194 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce3809dd40 con 0x7fce38135c80 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient: _finish_auth 0 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient: _check_auth_tickets 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:55.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 1 -- 172.21.15.64:0/1049940194 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fce2c001ae0 con 0x7fce38135c80 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce3f4e2640 5 monclient: authenticate success, global_id 124767 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce3f4e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient: dump: 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.226 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce3f4e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 1 -- 172.21.15.64:0/1049940194 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fce2c001c80 con 0x7fce38135c80 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 1 -- 172.21.15.64:0/1049940194 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fce2c004c20 con 0x7fce38135c80 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:55.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce37fff640 10 monclient: dump: 2024-09-17T15:51:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.228 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.229 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.229 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.229 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce3f4e2640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:55.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce3f4e2640 4 set_mon_vals no callback set 2024-09-17T15:51:55.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce3f4e2640 10 monclient: shutdown 2024-09-17T15:51:55.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce3f4e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:55.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce3f4e2640 1 -- 172.21.15.64:0/1049940194 >> v1:172.21.15.64:6789/0 conn(0x7fce38135c80 legacy=0x7fce3813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce3f4e2640 1 -- 172.21.15.64:0/1049940194 shutdown_connections 2024-09-17T15:51:55.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce3f4e2640 1 -- 172.21.15.64:0/1049940194 >> 172.21.15.64:0/1049940194 conn(0x7fce3806daa0 msgr2=0x7fce380ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:55.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce3f4e2640 1 -- 172.21.15.64:0/1049940194 shutdown_connections 2024-09-17T15:51:55.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.214+0000 7fce3f4e2640 1 -- 172.21.15.64:0/1049940194 wait complete. 2024-09-17T15:51:55.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.215+0000 7fce3f4e2640 10 monclient: build_initial_monmap 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.215+0000 7fce3f4e2640 10 monclient: monmap: 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:55.230 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:55.231 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:55.231 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:55.231 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.215+0000 7fce3f4e2640 1 Processor -- start 2024-09-17T15:51:55.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.215+0000 7fce3f4e2640 1 -- start start 2024-09-17T15:51:55.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.215+0000 7fce3f4e2640 10 monclient: init 2024-09-17T15:51:55.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce3f4e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:55.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce3f4e2640 10 monclient: _add_conns ranks=[5,7,6,1,8,4,3,0,2] 2024-09-17T15:51:55.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce3f4e2640 10 monclient(hunting): picked mon.noname-h con 0x7fce380a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:55.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce3f4e2640 10 monclient(hunting): picked mon.noname-i con 0x7fce38135c80 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:55.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce3f4e2640 10 monclient(hunting): picked mon.noname-d con 0x7fce381360a0 addr v1:172.21.15.64:6792/0 2024-09-17T15:51:55.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce3f4e2640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce380a2240 con 0x7fce380a54b0 2024-09-17T15:51:55.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce3f4e2640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce380a2520 con 0x7fce381360a0 2024-09-17T15:51:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce3f4e2640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce380a11b0 con 0x7fce38135c80 2024-09-17T15:51:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce3f4e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce3f4e2640 10 monclient(hunting): authenticate will time out at 1112.891446s 2024-09-17T15:51:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce3da7f640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fce381360a0 0x7fce381d5e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:43804/0 (socket says 172.21.15.64:43804) 2024-09-17T15:51:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce3da7f640 1 -- 172.21.15.64:0/84296819 learned_addr learned my addr 172.21.15.64:0/84296819 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2112085709 0 0) 0x7fce380a2520 con 0x7fce381360a0 2024-09-17T15:51:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce35ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce35ffb640 10 monclient(hunting): my global_id is 124764 2024-09-17T15:51:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.216+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce24003520 con 0x7fce381360a0 2024-09-17T15:51:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 278237543 0 0) 0x7fce24003520 con 0x7fce381360a0 2024-09-17T15:51:55.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce380a2520 con 0x7fce381360a0 2024-09-17T15:51:55.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fce28001a90 con 0x7fce381360a0 2024-09-17T15:51:55.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:55.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:51:55.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient(hunting): dump: 2024-09-17T15:51:55.233 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.233 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.233 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.233 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.233 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.233 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.233 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.233 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.233 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3010514958 0 0) 0x7fce380a2240 con 0x7fce380a54b0 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient(hunting): my global_id is 154184 2024-09-17T15:51:55.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce24003520 con 0x7fce380a54b0 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 957929970 0 0) 0x7fce380a11b0 con 0x7fce38135c80 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient(hunting): my global_id is 154195 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce380a2240 con 0x7fce38135c80 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1090386883 0 0) 0x7fce380a2520 con 0x7fce381360a0 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 >> v1:172.21.15.100:6792/0 conn(0x7fce38135c80 legacy=0x7fce3809f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 >> v1:172.21.15.100:6791/0 conn(0x7fce380a54b0 legacy=0x7fce3809ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 monclient: found mon.d 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce3809fb90 con 0x7fce381360a0 2024-09-17T15:51:55.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient: _finish_auth 0 2024-09-17T15:51:55.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:51:55.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:55.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce3f4e2640 5 monclient: authenticate success, global_id 124764 2024-09-17T15:51:55.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce3f4e2640 10 monclient: _renew_subs 2024-09-17T15:51:55.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce3f4e2640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:51:55.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce3f4e2640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fce3809fe40 con 0x7fce381360a0 2024-09-17T15:51:55.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fce280038c0 con 0x7fce381360a0 2024-09-17T15:51:55.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:55.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fce28003e70 con 0x7fce381360a0 2024-09-17T15:51:55.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:51:55.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce35ffb640 10 monclient: dump: 2024-09-17T15:51:55.236 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.237 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.237 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.237 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.237 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.237 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.237 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.237 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.237 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.237 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.237 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.237 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.237 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.237 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.238 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.238 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce3f4e2640 10 monclient: _renew_subs 2024-09-17T15:51:55.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce3f4e2640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:51:55.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce3f4e2640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fce381d7720 con 0x7fce381360a0 2024-09-17T15:51:55.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.217+0000 7fce377fe640 4 set_mon_vals no callback set 2024-09-17T15:51:55.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.218+0000 7fce3f4e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:55.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.218+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fce28004010 con 0x7fce381360a0 2024-09-17T15:51:55.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.218+0000 7fce3f4e2640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fce38077070 con 0x7fce3812f4f0 2024-09-17T15:51:55.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.218+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(285..285 src has 192..285) ==== 3506+0+0 (unknown 2220307653 0 0) 0x7fce2804ea20 con 0x7fce381360a0 2024-09-17T15:51:55.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.218+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3890493544 0 0) 0x7fce24003520 con 0x7fce3812f4f0 2024-09-17T15:51:55.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.218+0000 7fce35ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.218+0000 7fce35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.218+0000 7fce35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.218+0000 7fce35ffb640 10 monclient(hunting): my global_id is 154193 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.218+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce380a2520 con 0x7fce3812f4f0 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.218+0000 7fce3ca7d640 10 monclient: get_auth_request con 0x7fce24040580 auth_method 0 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.219+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3524909654 0 0) 0x7fce380a2520 con 0x7fce3812f4f0 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.219+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce24003520 con 0x7fce3812f4f0 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.219+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fce30001430 con 0x7fce3812f4f0 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.219+0000 7fce35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.219+0000 7fce35ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.219+0000 7fce35ffb640 10 monclient: dump: 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.239 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.219+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3787873908 0 0) 0x7fce24003520 con 0x7fce3812f4f0 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.219+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.100:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce380a74f0 con 0x7fce3812f4f0 2024-09-17T15:51:55.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.219+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fce30001740 con 0x7fce3812f4f0 2024-09-17T15:51:55.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.219+0000 7fce35ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:55.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.219+0000 7fce35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:55.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.219+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 >> v1:172.21.15.100:6791/0 conn(0x7fce3812f4f0 legacy=0x7fce3812f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.219+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 >> v1:172.21.15.100:6791/0 conn(0x7fce3812f4f0 legacy=0x7fce3812f8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:55.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.225+0000 7fce3f4e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:55.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.225+0000 7fce3f4e2640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fce380a6fc0 con 0x7fce3812ffe0 2024-09-17T15:51:55.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2013174145 0 0) 0x7fce380a6fc0 con 0x7fce3812ffe0 2024-09-17T15:51:55.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 10 monclient(hunting): my global_id is 154202 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce24003520 con 0x7fce3812ffe0 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2729512166 0 0) 0x7fce24003520 con 0x7fce3812ffe0 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce380a6fc0 con 0x7fce3812ffe0 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fce30004d10 con 0x7fce3812ffe0 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 10 monclient: dump: 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.243 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1335272919 0 0) 0x7fce380a6fc0 con 0x7fce3812ffe0 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.226+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 --> v1:172.21.15.100:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fce38068b30 con 0x7fce3812ffe0 2024-09-17T15:51:55.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.227+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 2208524678) 0x7fce300061d0 con 0x7fce3812ffe0 2024-09-17T15:51:55.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.227+0000 7fce35ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:55.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.227+0000 7fce35ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:55.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.227+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 >> v1:172.21.15.100:6791/0 conn(0x7fce3812ffe0 legacy=0x7fce381db910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.227+0000 7fce35ffb640 1 -- 172.21.15.64:0/84296819 >> v1:172.21.15.100:6791/0 conn(0x7fce3812ffe0 legacy=0x7fce381db910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:55.245 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:55.245 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:55.245 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:55.245 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:55.245 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:55.245 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 6569, 2024-09-17T15:51:55.245 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:55.246 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:55.246 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:55.246 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:55.246 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:55.246 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:55.246 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:55.246 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:55.246 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:55.246 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:55.246 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:55.246 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:51:55.246 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:55.246 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:55.247 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:55.248 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:55.248 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:55.248 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:55.248 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:55.248 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:55.248 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:55.248 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:55.248 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:55.248 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:55.248 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:55.248 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:55.248 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:55.248 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:55.249 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:55.250 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:55.251 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:55.251 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:55.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.251 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:55.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.251 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:55.251 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:55.251 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.251 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.251 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.251 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:55.251 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.252 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.252 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:55.252 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:55.252 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.252 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.252 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.252 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:55.252 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:55.252 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:55.255 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.255 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.255 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.255 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:55.255 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:55.255 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.255 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.255 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.255 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:55.256 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.256 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.256 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:55.256 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:55.256 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.256 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.256 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.256 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:55.256 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:55.257 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.257 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.257 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.257 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:55.257 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.257 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:55.257 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:55.257 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.257 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.259 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:55.260 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:55.262 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.263 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:55.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:55.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.264 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:55.264 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:55.264 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:55.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.264 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.264 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:55.264 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:55.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.227+0000 7fce3f4e2640 1 -- 172.21.15.64:0/84296819 >> v1:172.21.15.100:6812/4210963484 conn(0x7fce24040580 legacy=0x7fce24042a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.227+0000 7fce3f4e2640 10 monclient: shutdown 2024-09-17T15:51:55.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.227+0000 7fce3f4e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:55.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.227+0000 7fce3f4e2640 1 -- 172.21.15.64:0/84296819 >> v1:172.21.15.64:6792/0 conn(0x7fce381360a0 legacy=0x7fce381d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.227+0000 7fce3da7f640 1 -- 172.21.15.64:0/84296819 reap_dead start 2024-09-17T15:51:55.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.227+0000 7fce3da7f640 1 -- 172.21.15.64:0/84296819 reap_dead start 2024-09-17T15:51:55.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.227+0000 7fce3f4e2640 1 -- 172.21.15.64:0/84296819 shutdown_connections 2024-09-17T15:51:55.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.227+0000 7fce3f4e2640 1 -- 172.21.15.64:0/84296819 >> 172.21.15.64:0/84296819 conn(0x7fce3806daa0 msgr2=0x7fce380a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:55.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.228+0000 7fce3f4e2640 1 -- 172.21.15.64:0/84296819 shutdown_connections 2024-09-17T15:51:55.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.228+0000 7fce3f4e2640 1 -- 172.21.15.64:0/84296819 wait complete. 2024-09-17T15:51:55.266 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:55.266 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-17T15:51:55.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-17T15:51:55.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.367+0000 7f4142d8a640 10 monclient: get_monmap_and_config 2024-09-17T15:51:55.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.367+0000 7f4142d8a640 10 monclient: build_initial_monmap 2024-09-17T15:51:55.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.367+0000 7f4142d8a640 10 monclient: monmap: 2024-09-17T15:51:55.369 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:55.369 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.369 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:55.368258+0000 2024-09-17T15:51:55.369 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:55.368258+0000 2024-09-17T15:51:55.369 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:55.369 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.370 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:55.370 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:55.370 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:55.370 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:55.370 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:55.370 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:55.370 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:55.370 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:55.370 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:55.370 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4142d8a640 1 Processor -- start 2024-09-17T15:51:55.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4142d8a640 1 -- start start 2024-09-17T15:51:55.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4142d8a640 10 monclient: init 2024-09-17T15:51:55.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4142d8a640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:55.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4142d8a640 10 monclient: _add_conns ranks=[0,3,6,8,2,5,4,7,1] 2024-09-17T15:51:55.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4142d8a640 10 monclient(hunting): picked mon.noname-a con 0x7f413c0a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:55.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4142d8a640 10 monclient(hunting): picked mon.noname-g con 0x7f413c135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:55.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4142d8a640 10 monclient(hunting): picked mon.noname-d con 0x7f413c135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:51:55.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4142d8a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f413c09d6f0 con 0x7f413c0a5520 2024-09-17T15:51:55.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4142d8a640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f413c09d9d0 con 0x7f413c135b50 2024-09-17T15:51:55.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4142d8a640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f413c09dcb0 con 0x7f413c135f70 2024-09-17T15:51:55.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4142d8a640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:55.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4142d8a640 10 monclient(hunting): authenticate will time out at 1113.043741s 2024-09-17T15:51:55.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4140b26640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f413c0a5520 0x7f413c0a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:42578/0 (socket says 172.21.15.64:42578) 2024-09-17T15:51:55.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.368+0000 7f4140b26640 1 -- 172.21.15.64:0/461541227 learned_addr learned my addr 172.21.15.64:0/461541227 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:55.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 -- 172.21.15.64:0/461541227 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2094365601 0 0) 0x7f413c09dcb0 con 0x7f413c135f70 2024-09-17T15:51:55.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient(hunting): my global_id is 124773 2024-09-17T15:51:55.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 -- 172.21.15.64:0/461541227 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f412c003520 con 0x7f413c135f70 2024-09-17T15:51:55.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 -- 172.21.15.64:0/461541227 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1028305994 0 0) 0x7f413c09d6f0 con 0x7f413c0a5520 2024-09-17T15:51:55.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient(hunting): my global_id is 124776 2024-09-17T15:51:55.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 -- 172.21.15.64:0/461541227 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f413c09dcb0 con 0x7f413c0a5520 2024-09-17T15:51:55.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 -- 172.21.15.64:0/461541227 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3711841118 0 0) 0x7f413c09d9d0 con 0x7f413c135b50 2024-09-17T15:51:55.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient(hunting): my global_id is 115518 2024-09-17T15:51:55.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 -- 172.21.15.64:0/461541227 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f413c09d6f0 con 0x7f413c135b50 2024-09-17T15:51:55.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 -- 172.21.15.64:0/461541227 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2329790644 0 0) 0x7f412c003520 con 0x7f413c135f70 2024-09-17T15:51:55.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 -- 172.21.15.64:0/461541227 >> v1:172.21.15.100:6790/0 conn(0x7f413c135b50 legacy=0x7f413c13deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 -- 172.21.15.64:0/461541227 >> v1:172.21.15.64:6789/0 conn(0x7f413c0a5520 legacy=0x7f413c0a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:51:55.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 monclient: found mon.noname-d 2024-09-17T15:51:55.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:55.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:51:55.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 -- 172.21.15.64:0/461541227 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413c09ccd0 con 0x7f413c135f70 2024-09-17T15:51:55.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:55.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:55.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:55.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 -- 172.21.15.64:0/461541227 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4134001bb0 con 0x7f413c135f70 2024-09-17T15:51:55.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f4142d8a640 5 monclient: authenticate success, global_id 124773 2024-09-17T15:51:55.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f4142d8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:55.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:51:55.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient: dump: 2024-09-17T15:51:55.384 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.385 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.386 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f4142d8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 -- 172.21.15.64:0/461541227 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4134001d50 con 0x7f413c135f70 2024-09-17T15:51:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 1 -- 172.21.15.64:0/461541227 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4134003cd0 con 0x7f413c135f70 2024-09-17T15:51:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:51:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f413b7fe640 10 monclient: dump: 2024-09-17T15:51:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.386 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f4142d8a640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:55.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f4142d8a640 4 set_mon_vals no callback set 2024-09-17T15:51:55.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f4142d8a640 10 monclient: shutdown 2024-09-17T15:51:55.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f4142d8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:55.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.369+0000 7f4142d8a640 1 -- 172.21.15.64:0/461541227 >> v1:172.21.15.64:6792/0 conn(0x7f413c135f70 legacy=0x7f413c1415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.370+0000 7f4142d8a640 1 -- 172.21.15.64:0/461541227 shutdown_connections 2024-09-17T15:51:55.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.370+0000 7f4142d8a640 1 -- 172.21.15.64:0/461541227 >> 172.21.15.64:0/461541227 conn(0x7f413c06db30 msgr2=0x7f413c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:55.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.370+0000 7f4142d8a640 1 -- 172.21.15.64:0/461541227 shutdown_connections 2024-09-17T15:51:55.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.370+0000 7f4142d8a640 1 -- 172.21.15.64:0/461541227 wait complete. 2024-09-17T15:51:55.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.370+0000 7f4142d8a640 10 monclient: build_initial_monmap 2024-09-17T15:51:55.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.370+0000 7f4142d8a640 10 monclient: monmap: 2024-09-17T15:51:55.388 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:55.388 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:55.388 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:55.389 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:55.389 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.370+0000 7f4142d8a640 1 Processor -- start 2024-09-17T15:51:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4142d8a640 1 -- start start 2024-09-17T15:51:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4142d8a640 10 monclient: init 2024-09-17T15:51:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4142d8a640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4142d8a640 10 monclient: _add_conns ranks=[2,7,0,4,6,3,8,1,5] 2024-09-17T15:51:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4142d8a640 10 monclient(hunting): picked mon.noname-b con 0x7f413c0a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4142d8a640 10 monclient(hunting): picked mon.noname-i con 0x7f413c135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4142d8a640 10 monclient(hunting): picked mon.noname-a con 0x7f413c135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4142d8a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f413c13acc0 con 0x7f413c135f70 2024-09-17T15:51:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4142d8a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f413c0a63e0 con 0x7f413c0a5520 2024-09-17T15:51:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4142d8a640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f413c0a8fe0 con 0x7f413c135b50 2024-09-17T15:51:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4142d8a640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:55.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4142d8a640 10 monclient(hunting): authenticate will time out at 1113.046670s 2024-09-17T15:51:55.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4140b26640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f413c0a5520 0x7f413c137060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51426/0 (socket says 172.21.15.64:51426) 2024-09-17T15:51:55.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f4140b26640 1 -- 172.21.15.64:0/2051172147 learned_addr learned my addr 172.21.15.64:0/2051172147 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:55.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4126351359 0 0) 0x7f413c13acc0 con 0x7f413c135f70 2024-09-17T15:51:55.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f41397fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f41397fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f41397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f41397fa640 10 monclient(hunting): my global_id is 124785 2024-09-17T15:51:55.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4120003520 con 0x7f413c135f70 2024-09-17T15:51:55.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1628409678 0 0) 0x7f413c0a63e0 con 0x7f413c0a5520 2024-09-17T15:51:55.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f41397fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f41397fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f41397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f41397fa640 10 monclient(hunting): my global_id is 124859 2024-09-17T15:51:55.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.371+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f413c13acc0 con 0x7f413c0a5520 2024-09-17T15:51:55.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2320621128 0 0) 0x7f413c0a8fe0 con 0x7f413c135b50 2024-09-17T15:51:55.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): my global_id is 154204 2024-09-17T15:51:55.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f413c0a63e0 con 0x7f413c135b50 2024-09-17T15:51:55.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4213509458 0 0) 0x7f4120003520 con 0x7f413c135f70 2024-09-17T15:51:55.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f413c0a8fe0 con 0x7f413c135f70 2024-09-17T15:51:55.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4134001af0 con 0x7f413c135f70 2024-09-17T15:51:55.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:55.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): dump: 2024-09-17T15:51:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.393 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.394 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.394 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.394 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.394 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:55.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 452546370 0 0) 0x7f413c13acc0 con 0x7f413c0a5520 2024-09-17T15:51:55.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4120003520 con 0x7f413c0a5520 2024-09-17T15:51:55.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4130001b10 con 0x7f413c0a5520 2024-09-17T15:51:55.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:55.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:55.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): dump: 2024-09-17T15:51:55.395 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.395 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.395 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.395 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.395 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.395 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.396 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.396 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.396 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.396 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.396 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.396 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.396 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.396 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.396 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.396 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1556773149 0 0) 0x7f413c0a63e0 con 0x7f413c135b50 2024-09-17T15:51:55.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f413c13acc0 con 0x7f413c135b50 2024-09-17T15:51:55.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f41280019d0 con 0x7f413c135b50 2024-09-17T15:51:55.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:55.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:51:55.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient(hunting): dump: 2024-09-17T15:51:55.397 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.397 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.397 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.397 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.397 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.397 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.397 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.397 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.397 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.398 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.398 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.398 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.398 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.398 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.398 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.398 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 832212361 0 0) 0x7f413c0a8fe0 con 0x7f413c135f70 2024-09-17T15:51:55.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 >> v1:172.21.15.100:6792/0 conn(0x7f413c135b50 legacy=0x7f413c137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 >> v1:172.21.15.64:6790/0 conn(0x7f413c0a5520 legacy=0x7f413c137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient: _finish_hunting 0 2024-09-17T15:51:55.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 monclient: found mon.a 2024-09-17T15:51:55.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:55.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:51:55.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413c138960 con 0x7f413c135f70 2024-09-17T15:51:55.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient: _finish_auth 0 2024-09-17T15:51:55.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient: _check_auth_tickets 2024-09-17T15:51:55.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:55.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f4142d8a640 5 monclient: authenticate success, global_id 124785 2024-09-17T15:51:55.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f4142d8a640 10 monclient: _renew_subs 2024-09-17T15:51:55.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f4142d8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:51:55.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f4142d8a640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f413c13a420 con 0x7f413c135f70 2024-09-17T15:51:55.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f4142d8a640 10 monclient: _renew_subs 2024-09-17T15:51:55.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f4142d8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:51:55.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f4142d8a640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f413c142310 con 0x7f413c135f70 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4134003c20 con 0x7f413c135f70 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f41397fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.372+0000 7f413affd640 4 set_mon_vals no callback set 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4134004210 con 0x7f413c135f70 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f41397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f41397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f41397fa640 10 monclient: dump: 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.400 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.401 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.401 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.401 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.401 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.401 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.401 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.401 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.401 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.401 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f4142d8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:55.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f4142d8a640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f411c0065c0 con 0x7f411c002860 2024-09-17T15:51:55.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f4134004430 con 0x7f413c135f70 2024-09-17T15:51:55.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(285..285 src has 192..285) ==== 3506+0+0 (unknown 2220307653 0 0) 0x7f413404d8d0 con 0x7f413c135f70 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3148376612 0 0) 0x7f411c0065c0 con 0x7f411c002860 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f41397fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f41397fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f41397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f41397fa640 10 monclient(hunting): my global_id is 154213 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f413c0a8fe0 con 0x7f411c002860 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.373+0000 7f413bfff640 10 monclient: get_auth_request con 0x7f4120040c40 auth_method 0 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.374+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2962164839 0 0) 0x7f413c0a8fe0 con 0x7f411c002860 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.374+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f411c0065c0 con 0x7f411c002860 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.374+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4130001d40 con 0x7f411c002860 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.374+0000 7f41397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.374+0000 7f41397fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.374+0000 7f41397fa640 10 monclient: dump: 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.402 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.403 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.404 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.374+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3428758448 0 0) 0x7f411c0065c0 con 0x7f411c002860 2024-09-17T15:51:55.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.374+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.100:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f411c00a160 con 0x7f411c002860 2024-09-17T15:51:55.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.374+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4130004f70 con 0x7f411c002860 2024-09-17T15:51:55.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.374+0000 7f41397fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:55.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.374+0000 7f41397fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:55.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.374+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 >> v1:172.21.15.100:6792/0 conn(0x7f411c002860 legacy=0x7f411c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.374+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 >> v1:172.21.15.100:6792/0 conn(0x7f411c002860 legacy=0x7f411c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:55.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.380+0000 7f4142d8a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:55.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.380+0000 7f4142d8a640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f411c009e20 con 0x7f411c0090e0 2024-09-17T15:51:55.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1979508028 0 0) 0x7f411c009e20 con 0x7f411c0090e0 2024-09-17T15:51:55.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 10 monclient(hunting): my global_id is 154222 2024-09-17T15:51:55.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f411c0065c0 con 0x7f411c0090e0 2024-09-17T15:51:55.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3561495926 0 0) 0x7f411c0065c0 con 0x7f411c0090e0 2024-09-17T15:51:55.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f411c009e20 con 0x7f411c0090e0 2024-09-17T15:51:55.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f4130005710 con 0x7f411c0090e0 2024-09-17T15:51:55.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:51:55.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 10 monclient: dump: 2024-09-17T15:51:55.407 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.407 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.407 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.407 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.407 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.407 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.407 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.407 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.407 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.408 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.408 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.408 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.408 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.408 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.408 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.408 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3682783655 0 0) 0x7f411c009e20 con 0x7f411c0090e0 2024-09-17T15:51:55.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.381+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 --> v1:172.21.15.100:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f411c00c3f0 con 0x7f411c0090e0 2024-09-17T15:51:55.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 151602669) 0x7f4130006b60 con 0x7f411c0090e0 2024-09-17T15:51:55.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f41397fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:55.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f41397fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:55.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 >> v1:172.21.15.100:6792/0 conn(0x7f411c0090e0 legacy=0x7f411c009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f41397fa640 1 -- 172.21.15.64:0/2051172147 >> v1:172.21.15.100:6792/0 conn(0x7f411c0090e0 legacy=0x7f411c009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:55.409 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:55.409 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:55.409 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:55.409 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:55.409 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:55.409 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 6728, 2024-09-17T15:51:55.409 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:55.410 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:55.410 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:55.410 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:55.410 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:55.410 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:55.410 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:55.410 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:55.410 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:55.410 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:55.410 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:55.410 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:51:55.410 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:55.410 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:55.411 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:55.411 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:55.411 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:55.411 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:55.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:55.411 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:55.411 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:55.411 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:55.411 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:55.411 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:55.411 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:55.411 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:55.412 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:55.412 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:55.412 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:55.412 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:55.412 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:55.412 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:55.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:55.412 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:55.412 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:55.412 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:55.412 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:55.413 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:55.413 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:55.413 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:55.413 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.413 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:55.413 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:55.413 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:55.413 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:55.413 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:55.413 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:55.413 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:55.414 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:55.414 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:55.414 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:55.414 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:55.414 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:55.414 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:55.414 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:55.414 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:55.414 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:55.414 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:55.414 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:55.414 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:55.414 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:55.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:55.415 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:55.415 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:55.415 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:55.415 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:55.415 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:55.415 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:55.415 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:55.415 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:55.415 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:55.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.415 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:55.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.416 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:55.416 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:55.416 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.416 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.416 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.416 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:55.416 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.416 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.417 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:55.417 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:55.417 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.417 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.417 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.417 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:55.417 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:55.417 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.417 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.417 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.418 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:55.418 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.418 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:55.418 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:55.418 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.418 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.418 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.418 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.419 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.420 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.420 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:55.420 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:55.420 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.420 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.420 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:55.420 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.420 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.421 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:55.422 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:55.423 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.423 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.423 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.423 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:55.423 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.423 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.423 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:55.423 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:55.423 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.423 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.424 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.424 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:55.424 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:55.424 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.424 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.424 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.424 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:55.424 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.425 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.425 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:55.425 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:55.425 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.425 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.425 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.426 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:55.426 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:55.426 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.426 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.426 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:55.426 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.426 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.426 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:55.427 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:55.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.428 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.428 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:55.428 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:55.428 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:55.428 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:55.428 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:55.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.428 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.428 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:55.428 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:55.429 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:55.430 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:55.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f4142d8a640 1 -- 172.21.15.64:0/2051172147 >> v1:172.21.15.100:6812/4210963484 conn(0x7f4120040c40 legacy=0x7f4120043100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f4142d8a640 10 monclient: shutdown 2024-09-17T15:51:55.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f4142d8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:55.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f4142d8a640 1 -- 172.21.15.64:0/2051172147 >> v1:172.21.15.64:6789/0 conn(0x7f413c135f70 legacy=0x7f413c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f4141327640 1 -- 172.21.15.64:0/2051172147 reap_dead start 2024-09-17T15:51:55.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f4141327640 1 -- 172.21.15.64:0/2051172147 reap_dead start 2024-09-17T15:51:55.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f4142d8a640 1 -- 172.21.15.64:0/2051172147 shutdown_connections 2024-09-17T15:51:55.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f4142d8a640 1 -- 172.21.15.64:0/2051172147 >> 172.21.15.64:0/2051172147 conn(0x7f413c06db30 msgr2=0x7f413c0a4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:55.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f4142d8a640 1 -- 172.21.15.64:0/2051172147 shutdown_connections 2024-09-17T15:51:55.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.382+0000 7f4142d8a640 1 -- 172.21.15.64:0/2051172147 wait complete. 2024-09-17T15:51:55.432 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:55.432 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:51:55.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:51:55.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.531+0000 7fa6390bf640 10 monclient: get_monmap_and_config 2024-09-17T15:51:55.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.531+0000 7fa6390bf640 10 monclient: build_initial_monmap 2024-09-17T15:51:55.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.531+0000 7fa6390bf640 10 monclient: monmap: 2024-09-17T15:51:55.533 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:55.534 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.534 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:55.532696+0000 2024-09-17T15:51:55.534 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:55.532696+0000 2024-09-17T15:51:55.534 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:55.534 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.534 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:55.534 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:55.534 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:55.534 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:55.534 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:55.534 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:55.534 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:55.534 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:55.535 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:55.535 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.532+0000 7fa6390bf640 1 Processor -- start 2024-09-17T15:51:55.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.532+0000 7fa6390bf640 1 -- start start 2024-09-17T15:51:55.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.532+0000 7fa6390bf640 10 monclient: init 2024-09-17T15:51:55.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:55.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 10 monclient: _add_conns ranks=[8,7,5,4,3,0,2,6,1] 2024-09-17T15:51:55.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 10 monclient(hunting): picked mon.noname-e con 0x7fa6340a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:51:55.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 10 monclient(hunting): picked mon.noname-i con 0x7fa634135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:55.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 10 monclient(hunting): picked mon.noname-h con 0x7fa634135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:55.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa63409d6f0 con 0x7fa634135f70 2024-09-17T15:51:55.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa63409d9d0 con 0x7fa634135b50 2024-09-17T15:51:55.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa63409dcb0 con 0x7fa6340a5520 2024-09-17T15:51:55.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:55.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 10 monclient(hunting): authenticate will time out at 1113.208149s 2024-09-17T15:51:55.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa632d9d640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fa6340a5520 0x7fa6340a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:37064/0 (socket says 172.21.15.64:37064) 2024-09-17T15:51:55.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa632d9d640 1 -- 172.21.15.64:0/1322438009 learned_addr learned my addr 172.21.15.64:0/1322438009 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:55.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 1 -- 172.21.15.64:0/1322438009 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3221429118 0 0) 0x7fa63409dcb0 con 0x7fa6340a5520 2024-09-17T15:51:55.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient(hunting): my global_id is 154160 2024-09-17T15:51:55.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 1 -- 172.21.15.64:0/1322438009 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa620003520 con 0x7fa6340a5520 2024-09-17T15:51:55.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 1 -- 172.21.15.64:0/1322438009 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 668885954 0 0) 0x7fa620003520 con 0x7fa6340a5520 2024-09-17T15:51:55.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 1 -- 172.21.15.64:0/1322438009 >> v1:172.21.15.100:6792/0 conn(0x7fa634135b50 legacy=0x7fa63413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 1 -- 172.21.15.64:0/1322438009 >> v1:172.21.15.100:6791/0 conn(0x7fa634135f70 legacy=0x7fa6341415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient: _finish_hunting 0 2024-09-17T15:51:55.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 1 monclient: found mon.noname-e 2024-09-17T15:51:55.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:55.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:51:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 1 -- 172.21.15.64:0/1322438009 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63409ccd0 con 0x7fa6340a5520 2024-09-17T15:51:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient: _finish_auth 0 2024-09-17T15:51:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient: _check_auth_tickets 2024-09-17T15:51:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 5 monclient: authenticate success, global_id 154160 2024-09-17T15:51:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 1 -- 172.21.15.64:0/1322438009 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa624001830 con 0x7fa6340a5520 2024-09-17T15:51:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:51:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient: dump: 2024-09-17T15:51:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.544 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.545 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.545 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.545 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.545 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.545 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.545 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.545 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.545 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.545 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.545 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.545 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.545 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.545 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:55.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 1 -- 172.21.15.64:0/1322438009 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa6240019d0 con 0x7fa6340a5520 2024-09-17T15:51:55.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:55.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 1 -- 172.21.15.64:0/1322438009 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa624004be0 con 0x7fa6340a5520 2024-09-17T15:51:55.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:55.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa6390bf640 4 set_mon_vals no callback set 2024-09-17T15:51:55.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:51:55.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.533+0000 7fa63259c640 10 monclient: dump: 2024-09-17T15:51:55.546 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.546 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.546 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.534+0000 7fa6390bf640 10 monclient: shutdown 2024-09-17T15:51:55.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.534+0000 7fa6390bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:55.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.534+0000 7fa6390bf640 1 -- 172.21.15.64:0/1322438009 >> v1:172.21.15.64:6793/0 conn(0x7fa6340a5520 legacy=0x7fa6340a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.534+0000 7fa6390bf640 1 -- 172.21.15.64:0/1322438009 shutdown_connections 2024-09-17T15:51:55.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.534+0000 7fa6390bf640 1 -- 172.21.15.64:0/1322438009 >> 172.21.15.64:0/1322438009 conn(0x7fa63406db30 msgr2=0x7fa6340ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:55.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.534+0000 7fa6390bf640 1 -- 172.21.15.64:0/1322438009 shutdown_connections 2024-09-17T15:51:55.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.534+0000 7fa6390bf640 1 -- 172.21.15.64:0/1322438009 wait complete. 2024-09-17T15:51:55.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.535+0000 7fa6390bf640 10 monclient: build_initial_monmap 2024-09-17T15:51:55.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.535+0000 7fa6390bf640 10 monclient: monmap: 2024-09-17T15:51:55.548 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:55.548 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:55.548 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:55.549 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:55.549 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:55.549 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:55.549 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:55.549 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:55.549 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:55.549 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:55.549 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:55.549 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:55.550 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:55.550 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:55.550 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:55.550 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.535+0000 7fa6390bf640 1 Processor -- start 2024-09-17T15:51:55.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.535+0000 7fa6390bf640 1 -- start start 2024-09-17T15:51:55.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.535+0000 7fa6390bf640 10 monclient: init 2024-09-17T15:51:55.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.535+0000 7fa6390bf640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:55.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.535+0000 7fa6390bf640 10 monclient: _add_conns ranks=[0,1,8,2,6,5,3,4,7] 2024-09-17T15:51:55.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa6390bf640 10 monclient(hunting): picked mon.noname-a con 0x7fa6340a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:55.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa6390bf640 10 monclient(hunting): picked mon.noname-f con 0x7fa634135b50 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:55.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa6390bf640 10 monclient(hunting): picked mon.noname-e con 0x7fa634135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:51:55.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa6390bf640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6340a0aa0 con 0x7fa6340a5520 2024-09-17T15:51:55.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa6390bf640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa63409e740 con 0x7fa634135b50 2024-09-17T15:51:55.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa6390bf640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa634132b30 con 0x7fa634135f70 2024-09-17T15:51:55.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa6390bf640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:55.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa6390bf640 10 monclient(hunting): authenticate will time out at 1113.211115s 2024-09-17T15:51:55.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa632d9d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fa6340a5520 0x7fa634131d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:42596/0 (socket says 172.21.15.64:42596) 2024-09-17T15:51:55.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa63359e640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fa634135f70 0x7fa6341d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:37068/0 (socket says 172.21.15.64:37068) 2024-09-17T15:51:55.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa632d9d640 1 -- 172.21.15.64:0/2400452787 learned_addr learned my addr 172.21.15.64:0/2400452787 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:55.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1125473180 0 0) 0x7fa6340a0aa0 con 0x7fa6340a5520 2024-09-17T15:51:55.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): my global_id is 124794 2024-09-17T15:51:55.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa614003550 con 0x7fa6340a5520 2024-09-17T15:51:55.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3523550979 0 0) 0x7fa634132b30 con 0x7fa634135f70 2024-09-17T15:51:55.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): my global_id is 154169 2024-09-17T15:51:55.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6340a0aa0 con 0x7fa634135f70 2024-09-17T15:51:55.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4226916006 0 0) 0x7fa63409e740 con 0x7fa634135b50 2024-09-17T15:51:55.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:55.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:55.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:55.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): my global_id is 6589 2024-09-17T15:51:55.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa634132b30 con 0x7fa634135b50 2024-09-17T15:51:55.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1229362527 0 0) 0x7fa6340a0aa0 con 0x7fa634135f70 2024-09-17T15:51:55.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa63409e740 con 0x7fa634135f70 2024-09-17T15:51:55.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1308313291 0 0) 0x7fa614003550 con 0x7fa6340a5520 2024-09-17T15:51:55.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6340a0aa0 con 0x7fa6340a5520 2024-09-17T15:51:55.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa62c001c70 con 0x7fa634135f70 2024-09-17T15:51:55.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:55.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:51:55.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): dump: 2024-09-17T15:51:55.554 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.554 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.554 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.554 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.554 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.554 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.554 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.554 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.554 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.554 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.554 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.554 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.555 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.555 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.555 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.555 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:55.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa6240017f0 con 0x7fa6340a5520 2024-09-17T15:51:55.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:55.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:55.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): dump: 2024-09-17T15:51:55.555 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.555 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.555 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.555 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.556 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.556 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.556 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.556 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.556 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.556 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.556 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.556 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.556 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.556 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.556 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.556 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3702802374 0 0) 0x7fa634132b30 con 0x7fa634135b50 2024-09-17T15:51:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa614003550 con 0x7fa634135b50 2024-09-17T15:51:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa61c001a50 con 0x7fa634135b50 2024-09-17T15:51:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:51:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient(hunting): dump: 2024-09-17T15:51:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.558 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.558 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.558 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.558 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.558 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.558 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.558 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.558 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.558 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1559236994 0 0) 0x7fa63409e740 con 0x7fa634135f70 2024-09-17T15:51:55.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 >> v1:172.21.15.100:6789/0 conn(0x7fa634135b50 legacy=0x7fa634132420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 >> v1:172.21.15.64:6789/0 conn(0x7fa6340a5520 legacy=0x7fa634131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 monclient: found mon.e 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6340a0200 con 0x7fa634135f70 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa62b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa6390bf640 5 monclient: authenticate success, global_id 154169 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa6390bf640 10 monclient: _renew_subs 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa6390bf640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.536+0000 7fa6390bf640 1 -- 172.21.15.64:0/2400452787 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa6340a04e0 con 0x7fa634135f70 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.537+0000 7fa6390bf640 10 monclient: _renew_subs 2024-09-17T15:51:55.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.537+0000 7fa6390bf640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:55.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.537+0000 7fa6390bf640 1 -- 172.21.15.64:0/2400452787 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fa6340a5150 con 0x7fa634135f70 2024-09-17T15:51:55.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.537+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa62c003990 con 0x7fa634135f70 2024-09-17T15:51:55.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.537+0000 7fa62b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:55.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.537+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa62c003f40 con 0x7fa634135f70 2024-09-17T15:51:55.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.537+0000 7fa62b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:55.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.537+0000 7fa62b7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:51:55.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.537+0000 7fa631d9b640 4 set_mon_vals no callback set 2024-09-17T15:51:55.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.537+0000 7fa62b7fe640 10 monclient: dump: 2024-09-17T15:51:55.560 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:55.560 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:55.560 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.560 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:55.560 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:55.561 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:55.561 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:55.561 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:55.561 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:55.561 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:55.561 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:55.561 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:55.561 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:55.561 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:55.561 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:55.561 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:55.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.537+0000 7fa6390bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:55.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.538+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fa62c0040e0 con 0x7fa634135f70 2024-09-17T15:51:55.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.538+0000 7fa6390bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:55.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.538+0000 7fa6390bf640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:55.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.538+0000 7fa6390bf640 1 -- 172.21.15.64:0/2400452787 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa618005080 con 0x7fa634135f70 2024-09-17T15:51:55.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.538+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(285..285 src has 192..285) ==== 3506+0+0 (unknown 2220307653 0 0) 0x7fa62c04eba0 con 0x7fa634135f70 2024-09-17T15:51:55.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.538+0000 7fa632d9d640 10 monclient: get_auth_request con 0x7fa614040cd0 auth_method 0 2024-09-17T15:51:55.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.542+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.8 v1:172.21.15.64:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fa62c017790 con 0x7fa634135f70 2024-09-17T15:51:55.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.542+0000 7fa62b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:55.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.542+0000 7fa62b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:55.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.808+0000 7fa6390bf640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:51:55.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.808+0000 7fa6390bf640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:51:55.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.808+0000 7fa6390bf640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:55.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.808+0000 7fa6390bf640 1 -- 172.21.15.64:0/2400452787 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fa618005c40 con 0x7fa634135f70 2024-09-17T15:51:55.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.810+0000 7fa62b7fe640 1 -- 172.21.15.64:0/2400452787 <== mon.8 v1:172.21.15.64:6793/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fa62c01bb70 con 0x7fa634135f70 2024-09-17T15:51:55.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.810+0000 7fa62b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:51:55.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.810+0000 7fa62b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:55.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.813+0000 7fa6390bf640 1 -- 172.21.15.64:0/2400452787 >> v1:172.21.15.100:6812/4210963484 conn(0x7fa614040cd0 legacy=0x7fa614043190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.813+0000 7fa6390bf640 10 monclient: shutdown 2024-09-17T15:51:55.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.813+0000 7fa6390bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:55.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.813+0000 7fa6390bf640 1 -- 172.21.15.64:0/2400452787 >> v1:172.21.15.64:6793/0 conn(0x7fa634135f70 legacy=0x7fa6341d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:55.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.813+0000 7fa6390bf640 1 -- 172.21.15.64:0/2400452787 shutdown_connections 2024-09-17T15:51:55.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.813+0000 7fa6390bf640 1 -- 172.21.15.64:0/2400452787 >> 172.21.15.64:0/2400452787 conn(0x7fa63406db30 msgr2=0x7fa63412c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:55.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.813+0000 7fa6390bf640 1 -- 172.21.15.64:0/2400452787 shutdown_connections 2024-09-17T15:51:55.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:55.813+0000 7fa6390bf640 1 -- 172.21.15.64:0/2400452787 wait complete. 2024-09-17T15:51:55.832 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:51:55.856 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.856 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=1,health=12,kv=12,logm=54} crc {auth=2092428508,config=1747058506,health=590170703,kv=1246312931,logm=3057255489}) 2024-09-17T15:51:55.856 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=12,logm=51} crc {auth=2764548270,config=2586702516,health=590170703,kv=1246312931,logm=1729294078}) 2024-09-17T15:51:55.856 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.856 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=1,health=12,kv=12,logm=54} crc {auth=2092428508,config=1747058506,health=590170703,kv=1246312931,logm=3057255489}) 2024-09-17T15:51:55.857 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=12,logm=52} crc {auth=1769029924,config=2586702516,health=590170703,kv=1246312931,logm=3384215020}) 2024-09-17T15:51:55.857 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.857 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=1,health=12,kv=12,logm=54} crc {auth=2092428508,config=1747058506,health=590170703,kv=1246312931,logm=3057255489}) 2024-09-17T15:51:55.857 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=22,config=2,health=12,kv=11,logm=53} crc {auth=2849217072,config=2586702516,health=590170703,kv=4080546736,logm=4148047058}) 2024-09-17T15:51:55.857 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.857 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=1,health=12,kv=12,logm=54} crc {auth=2092428508,config=1747058506,health=590170703,kv=1246312931,logm=3057255489}) 2024-09-17T15:51:55.857 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=22,config=2,health=12,kv=10,logm=54} crc {auth=1783666433,config=2586702516,health=5765848,kv=3466664689,logm=2465928591}) 2024-09-17T15:51:55.857 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.857 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=1,health=12,kv=12,logm=54} crc {auth=2092428508,config=1747058506,health=590170703,kv=1246312931,logm=3057255489}) 2024-09-17T15:51:55.857 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=22,config=2,health=12,kv=11,logm=53} crc {auth=1292169179,config=2586702516,health=590170703,kv=1234106001,logm=2076128824}) 2024-09-17T15:51:55.857 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.857 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=1,health=12,kv=12,logm=54} crc {auth=2092428508,config=1747058506,health=590170703,kv=1246312931,logm=3057255489}) 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=21,config=2,health=10,kv=12,logm=55} crc {auth=1998002148,config=2586702516,health=1308591922,kv=1246312931,logm=4041097416}) 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=1,health=12,kv=12,logm=54} crc {auth=2092428508,config=1747058506,health=590170703,kv=1246312931,logm=3057255489}) 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=22,config=2,health=11,kv=12,logm=53} crc {auth=3723725973,config=2586702516,health=421010566,kv=1246312931,logm=3851274371}) 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=1,health=12,kv=12,logm=54} crc {auth=2092428508,config=1747058506,health=590170703,kv=1246312931,logm=3057255489}) 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.810+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=21,config=2,health=10,kv=12,logm=55} crc {auth=3124543449,config=2586702516,health=496074455,kv=4151102094,logm=3373008808}) 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240488642}) 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238754246}) 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240488642}) 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=962042765}) 2024-09-17T15:51:55.858 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.859 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240488642}) 2024-09-17T15:51:55.859 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=756227938}) 2024-09-17T15:51:55.859 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.859 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240488642}) 2024-09-17T15:51:55.859 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1884653371}) 2024-09-17T15:51:55.859 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.859 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240488642}) 2024-09-17T15:51:55.859 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2680930994}) 2024-09-17T15:51:55.859 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.859 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240488642}) 2024-09-17T15:51:55.859 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3834939911}) 2024-09-17T15:51:55.859 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.859 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240488642}) 2024-09-17T15:51:55.860 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4043461205}) 2024-09-17T15:51:55.860 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.860 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240488642}) 2024-09-17T15:51:55.860 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.811+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=856895634}) 2024-09-17T15:51:55.860 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.860 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690900009}) 2024-09-17T15:51:55.860 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1844829194}) 2024-09-17T15:51:55.860 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.860 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690900009}) 2024-09-17T15:51:55.860 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=230535691}) 2024-09-17T15:51:55.860 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.860 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690900009}) 2024-09-17T15:51:55.860 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4051006589}) 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690900009}) 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1385419528}) 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690900009}) 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1894314373}) 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690900009}) 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=274569195}) 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690900009}) 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1416069395}) 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690900009}) 2024-09-17T15:51:55.861 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.813+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2212575748}) 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1482433600}) 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590564306}) 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1482433600}) 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4164393593}) 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1482433600}) 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4231858536}) 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1482433600}) 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2321104050}) 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.862 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1482433600}) 2024-09-17T15:51:55.863 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=685577579}) 2024-09-17T15:51:55.863 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.863 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1482433600}) 2024-09-17T15:51:55.863 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3474297569}) 2024-09-17T15:51:55.863 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.863 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1482433600}) 2024-09-17T15:51:55.863 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=725511027}) 2024-09-17T15:51:55.863 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.863 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1482433600}) 2024-09-17T15:51:55.863 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.814+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2892625610}) 2024-09-17T15:51:55.863 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.863 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585127450}) 2024-09-17T15:51:55.863 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1467065111}) 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585127450}) 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3091613813}) 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585127450}) 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1135648926}) 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585127450}) 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4142546020}) 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585127450}) 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.815+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=798390771}) 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.816+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.864 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.816+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585127450}) 2024-09-17T15:51:55.865 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.816+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=227954069}) 2024-09-17T15:51:55.865 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.816+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.865 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.816+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585127450}) 2024-09-17T15:51:55.865 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.816+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1254673333}) 2024-09-17T15:51:55.865 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.816+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.865 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.816+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585127450}) 2024-09-17T15:51:55.865 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.816+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3024448181}) 2024-09-17T15:51:55.865 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.865 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778603615}) 2024-09-17T15:51:55.865 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3625741581}) 2024-09-17T15:51:55.865 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.865 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778603615}) 2024-09-17T15:51:55.865 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483208339}) 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778603615}) 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2268525207}) 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778603615}) 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=576492478}) 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778603615}) 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1481478423}) 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778603615}) 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3556493962}) 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.866 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778603615}) 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1622663881}) 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778603615}) 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.817+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2417838104}) 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649991163}) 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002811281}) 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649991163}) 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3947601746}) 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649991163}) 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1559060689}) 2024-09-17T15:51:55.867 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649991163}) 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1150525597}) 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649991163}) 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=89275609}) 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649991163}) 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1205522634}) 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649991163}) 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=724853542}) 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649991163}) 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.818+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3626271021}) 2024-09-17T15:51:55.868 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2038761045}) 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176744158}) 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2038761045}) 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642628170}) 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2038761045}) 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1596261703}) 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2038761045}) 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=758260998}) 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2038761045}) 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=698365516}) 2024-09-17T15:51:55.869 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.870 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2038761045}) 2024-09-17T15:51:55.870 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2468296487}) 2024-09-17T15:51:55.870 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.870 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2038761045}) 2024-09-17T15:51:55.870 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4155585922}) 2024-09-17T15:51:55.870 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.870 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2038761045}) 2024-09-17T15:51:55.870 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.820+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3781632614}) 2024-09-17T15:51:55.870 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.870 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855590593}) 2024-09-17T15:51:55.870 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235722135}) 2024-09-17T15:51:55.870 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.870 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855590593}) 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475739596}) 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855590593}) 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2399322982}) 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855590593}) 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1123022041}) 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855590593}) 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3760575462}) 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855590593}) 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2069184914}) 2024-09-17T15:51:55.871 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855590593}) 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2730685561}) 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2855590593}) 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.821+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3077078857}) 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545624964}) 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267930623}) 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545624964}) 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=145389597}) 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545624964}) 2024-09-17T15:51:55.872 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=163085097}) 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545624964}) 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2630459909}) 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545624964}) 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1213738720}) 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545624964}) 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2255553252}) 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545624964}) 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3534861130}) 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.873 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545624964}) 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.823+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=583896699}) 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110412301}) 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3887763842}) 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110412301}) 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3041013953}) 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110412301}) 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1834759411}) 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110412301}) 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4199164032}) 2024-09-17T15:51:55.874 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.875 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110412301}) 2024-09-17T15:51:55.875 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4117347627}) 2024-09-17T15:51:55.875 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.875 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110412301}) 2024-09-17T15:51:55.875 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=65296991}) 2024-09-17T15:51:55.875 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.875 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110412301}) 2024-09-17T15:51:55.875 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1873271492}) 2024-09-17T15:51:55.875 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.875 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110412301}) 2024-09-17T15:51:55.875 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.824+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3251908735}) 2024-09-17T15:51:55.875 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.825+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.876 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.825+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4143271956}) 2024-09-17T15:51:55.876 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.825+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205390957}) 2024-09-17T15:51:55.876 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.825+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.876 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.825+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4143271956}) 2024-09-17T15:51:55.876 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2194367711}) 2024-09-17T15:51:55.876 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.876 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4143271956}) 2024-09-17T15:51:55.876 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=446372977}) 2024-09-17T15:51:55.876 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.876 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4143271956}) 2024-09-17T15:51:55.876 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1161736770}) 2024-09-17T15:51:55.876 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.876 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4143271956}) 2024-09-17T15:51:55.877 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=647498985}) 2024-09-17T15:51:55.877 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.877 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4143271956}) 2024-09-17T15:51:55.877 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4170461131}) 2024-09-17T15:51:55.877 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.877 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4143271956}) 2024-09-17T15:51:55.877 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2912033727}) 2024-09-17T15:51:55.877 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.877 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4143271956}) 2024-09-17T15:51:55.877 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.826+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3536695536}) 2024-09-17T15:51:55.877 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.877 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3928599991}) 2024-09-17T15:51:55.877 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815175179}) 2024-09-17T15:51:55.878 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.878 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3928599991}) 2024-09-17T15:51:55.878 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=107134968}) 2024-09-17T15:51:55.878 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.878 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3928599991}) 2024-09-17T15:51:55.878 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=660471232}) 2024-09-17T15:51:55.878 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.878 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3928599991}) 2024-09-17T15:51:55.878 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1621603135}) 2024-09-17T15:51:55.878 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.878 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3928599991}) 2024-09-17T15:51:55.878 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3320803800}) 2024-09-17T15:51:55.878 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.879 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3928599991}) 2024-09-17T15:51:55.879 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3583846554}) 2024-09-17T15:51:55.879 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.879 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3928599991}) 2024-09-17T15:51:55.879 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2944825748}) 2024-09-17T15:51:55.879 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.879 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3928599991}) 2024-09-17T15:51:55.879 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.827+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2199074052}) 2024-09-17T15:51:55.879 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.879 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86053031}) 2024-09-17T15:51:55.879 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2244222851}) 2024-09-17T15:51:55.879 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.879 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86053031}) 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1117791676}) 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86053031}) 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1880632685}) 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86053031}) 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=758767979}) 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86053031}) 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=782511813}) 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86053031}) 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2967036246}) 2024-09-17T15:51:55.880 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86053031}) 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1070787151}) 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.828+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86053031}) 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.829+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2994215122}) 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447346649}) 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012606726}) 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447346649}) 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1812023712}) 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447346649}) 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2469055479}) 2024-09-17T15:51:55.881 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.882 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447346649}) 2024-09-17T15:51:55.882 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2050065913}) 2024-09-17T15:51:55.882 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.882 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447346649}) 2024-09-17T15:51:55.882 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1994247152}) 2024-09-17T15:51:55.882 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.882 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447346649}) 2024-09-17T15:51:55.882 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1527597879}) 2024-09-17T15:51:55.882 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.882 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447346649}) 2024-09-17T15:51:55.882 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1568123972}) 2024-09-17T15:51:55.882 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.882 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447346649}) 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.830+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3609260574}) 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=238342662}) 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946728250}) 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=238342662}) 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2344733244}) 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=238342662}) 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=435050598}) 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=238342662}) 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2350289158}) 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.883 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=238342662}) 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3875470511}) 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=238342662}) 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1904747594}) 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=238342662}) 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3394534365}) 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=238342662}) 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.831+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1792222049}) 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=498790571}) 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2404587790}) 2024-09-17T15:51:55.884 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.885 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=498790571}) 2024-09-17T15:51:55.885 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=339213476}) 2024-09-17T15:51:55.885 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.885 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=498790571}) 2024-09-17T15:51:55.885 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2780124036}) 2024-09-17T15:51:55.885 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.885 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=498790571}) 2024-09-17T15:51:55.885 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=35525505}) 2024-09-17T15:51:55.885 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.885 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=498790571}) 2024-09-17T15:51:55.885 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3886743663}) 2024-09-17T15:51:55.885 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.885 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=498790571}) 2024-09-17T15:51:55.886 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2306010177}) 2024-09-17T15:51:55.886 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.886 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=498790571}) 2024-09-17T15:51:55.886 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.832+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4078148771}) 2024-09-17T15:51:55.886 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.833+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.886 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.833+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=498790571}) 2024-09-17T15:51:55.886 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.833+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2958267063}) 2024-09-17T15:51:55.886 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.886 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086194692}) 2024-09-17T15:51:55.886 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1844415395}) 2024-09-17T15:51:55.886 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.886 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086194692}) 2024-09-17T15:51:55.886 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1822567727}) 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086194692}) 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2791754622}) 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086194692}) 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4216456576}) 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086194692}) 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1021759587}) 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086194692}) 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4207331930}) 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.887 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086194692}) 2024-09-17T15:51:55.888 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1863896764}) 2024-09-17T15:51:55.888 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.888 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086194692}) 2024-09-17T15:51:55.888 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.834+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=911771570}) 2024-09-17T15:51:55.888 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.888 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313181174}) 2024-09-17T15:51:55.888 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558204697}) 2024-09-17T15:51:55.888 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.888 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313181174}) 2024-09-17T15:51:55.888 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1524417754}) 2024-09-17T15:51:55.888 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.888 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313181174}) 2024-09-17T15:51:55.888 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2398775508}) 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313181174}) 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2639314505}) 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313181174}) 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3731530903}) 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313181174}) 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2607138151}) 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313181174}) 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2217668349}) 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.889 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313181174}) 2024-09-17T15:51:55.890 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.835+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1050491621}) 2024-09-17T15:51:55.890 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.890 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758420084}) 2024-09-17T15:51:55.890 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=205260315}) 2024-09-17T15:51:55.890 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.890 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758420084}) 2024-09-17T15:51:55.890 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2230781486}) 2024-09-17T15:51:55.890 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.890 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758420084}) 2024-09-17T15:51:55.890 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1258392527}) 2024-09-17T15:51:55.890 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.890 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758420084}) 2024-09-17T15:51:55.890 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1224947916}) 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758420084}) 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=198995234}) 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758420084}) 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=946961126}) 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758420084}) 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2799111318}) 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758420084}) 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.836+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2915578630}) 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.837+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.891 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.837+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3232524283}) 2024-09-17T15:51:55.892 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.837+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554372649}) 2024-09-17T15:51:55.892 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.837+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.892 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.837+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3232524283}) 2024-09-17T15:51:55.892 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.837+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4223584942}) 2024-09-17T15:51:55.892 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.837+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.892 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.837+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3232524283}) 2024-09-17T15:51:55.892 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.837+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3778753663}) 2024-09-17T15:51:55.892 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.837+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.892 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.837+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3232524283}) 2024-09-17T15:51:55.892 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.838+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3819547559}) 2024-09-17T15:51:55.892 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.838+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.892 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.838+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3232524283}) 2024-09-17T15:51:55.892 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.838+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2216774482}) 2024-09-17T15:51:55.893 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.838+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.893 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.838+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3232524283}) 2024-09-17T15:51:55.893 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.838+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3665779617}) 2024-09-17T15:51:55.893 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.838+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.893 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.838+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3232524283}) 2024-09-17T15:51:55.893 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.838+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=906457391}) 2024-09-17T15:51:55.893 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.838+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.893 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.838+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3232524283}) 2024-09-17T15:51:55.893 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.838+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=72598063}) 2024-09-17T15:51:55.893 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.894 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=1527909810,mdsmap=1462844619,mgr=2970304559,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=2187931326,monmap=1573330833,nvmeofgw=598337687,osd_metadata=4100752439,osd_pg_creating=907393130,osd_snap=723727393,osdmap=1996083082}) 2024-09-17T15:51:55.894 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=30,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_snap=2,osdmap=14} crc {logm=236333067,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=740524101,osd_snap=723727393,osdmap=3999762530}) 2024-09-17T15:51:55.894 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.894 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=1527909810,mdsmap=1462844619,mgr=2970304559,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=2187931326,monmap=1573330833,nvmeofgw=598337687,osd_metadata=4100752439,osd_pg_creating=907393130,osd_snap=723727393,osdmap=1996083082}) 2024-09-17T15:51:55.894 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=32,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=14} crc {logm=3069849939,mdsmap=1462844619,mgr=4116046384,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=3632003337,monmap=1573330833,nvmeofgw=598337686,osd_metadata=195458673,osd_pg_creating=907393130,osd_snap=723727393,osdmap=3999762530}) 2024-09-17T15:51:55.894 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.894 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=1527909810,mdsmap=1462844619,mgr=2970304559,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=2187931326,monmap=1573330833,nvmeofgw=598337687,osd_metadata=4100752439,osd_pg_creating=907393130,osd_snap=723727393,osdmap=1996083082}) 2024-09-17T15:51:55.894 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=32,mdsmap=3,mgr=5,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=15} crc {logm=3091550617,mdsmap=1462844619,mgr=1866405893,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=907393130,osd_snap=723727393,osdmap=2510352854}) 2024-09-17T15:51:55.894 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.894 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=1527909810,mdsmap=1462844619,mgr=2970304559,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=2187931326,monmap=1573330833,nvmeofgw=598337687,osd_metadata=4100752439,osd_pg_creating=907393130,osd_snap=723727393,osdmap=1996083082}) 2024-09-17T15:51:55.894 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=31,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=15} crc {logm=446440531,mdsmap=3196345284,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=3022855382,mon_config_key=2807123856,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=907393130,osd_snap=723727393,osdmap=2510352854}) 2024-09-17T15:51:55.894 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.894 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=1527909810,mdsmap=1462844619,mgr=2970304559,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=2187931326,monmap=1573330833,nvmeofgw=598337687,osd_metadata=4100752439,osd_pg_creating=907393130,osd_snap=723727393,osdmap=1996083082}) 2024-09-17T15:51:55.895 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=32,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=15} crc {logm=795436612,mdsmap=1462844619,mgr=4278719971,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=2892874311,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=907393130,osd_snap=723727393,osdmap=2510352854}) 2024-09-17T15:51:55.895 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.895 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=1527909810,mdsmap=1462844619,mgr=2970304559,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=2187931326,monmap=1573330833,nvmeofgw=598337687,osd_metadata=4100752439,osd_pg_creating=907393130,osd_snap=723727393,osdmap=1996083082}) 2024-09-17T15:51:55.895 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=31,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=13} crc {logm=1019639760,mdsmap=1462844619,mgr=2417410626,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=907393130,osd_snap=723727393,osdmap=1796542313}) 2024-09-17T15:51:55.895 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.895 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=1527909810,mdsmap=1462844619,mgr=2970304559,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=2187931326,monmap=1573330833,nvmeofgw=598337687,osd_metadata=4100752439,osd_pg_creating=907393130,osd_snap=723727393,osdmap=1996083082}) 2024-09-17T15:51:55.895 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=14} crc {logm=3853257917,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=109190375,mon_config_key=2137179104,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=907393130,osd_snap=723727393,osdmap=1952490351}) 2024-09-17T15:51:55.895 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.895 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=1527909810,mdsmap=1462844619,mgr=2970304559,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=2187931326,monmap=1573330833,nvmeofgw=598337687,osd_metadata=4100752439,osd_pg_creating=907393130,osd_snap=723727393,osdmap=1996083082}) 2024-09-17T15:51:55.895 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.839+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=30,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=15} crc {logm=4206175859,mdsmap=1462844619,mgr=4116046384,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=922168801,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=60438592,osd_pg_creating=907393130,osd_snap=723727393,osdmap=3155802903}) 2024-09-17T15:51:55.895 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.895 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=791254777}) 2024-09-17T15:51:55.896 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=997262108}) 2024-09-17T15:51:55.896 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.896 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=791254777}) 2024-09-17T15:51:55.896 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2860624267}) 2024-09-17T15:51:55.896 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.896 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=791254777}) 2024-09-17T15:51:55.896 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=646009156}) 2024-09-17T15:51:55.896 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.896 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=791254777}) 2024-09-17T15:51:55.896 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2201945847}) 2024-09-17T15:51:55.896 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.896 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=791254777}) 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2075464895}) 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=791254777}) 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2054507411}) 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=791254777}) 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=696770404}) 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=791254777}) 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.841+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3261540102}) 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=41} crc {osdmap=991729195}) 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=43} crc {osdmap=3731430298}) 2024-09-17T15:51:55.897 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.898 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=41} crc {osdmap=991729195}) 2024-09-17T15:51:55.898 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=43} crc {osdmap=1448093795}) 2024-09-17T15:51:55.898 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.898 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=41} crc {osdmap=991729195}) 2024-09-17T15:51:55.898 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=39} crc {osdmap=2267280407}) 2024-09-17T15:51:55.898 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.898 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=41} crc {osdmap=991729195}) 2024-09-17T15:51:55.898 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=41} crc {osdmap=2933778163}) 2024-09-17T15:51:55.898 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.898 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=41} crc {osdmap=991729195}) 2024-09-17T15:51:55.898 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=40} crc {osdmap=669739303}) 2024-09-17T15:51:55.898 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.899 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=41} crc {osdmap=991729195}) 2024-09-17T15:51:55.899 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=40} crc {osdmap=4126052577}) 2024-09-17T15:51:55.899 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.899 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=41} crc {osdmap=991729195}) 2024-09-17T15:51:55.899 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=42} crc {osdmap=1381066654}) 2024-09-17T15:51:55.899 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:55.899 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=41} crc {osdmap=991729195}) 2024-09-17T15:51:55.899 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:51:55.842+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=40} crc {osdmap=2925635222}) 2024-09-17T15:51:56.827 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:56.828 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39739]: starting. 2024-09-17T15:51:56.828 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39739]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39584 2024-09-17T15:51:56.828 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39739]: created object 0... 2024-09-17T15:51:56.828 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39739]: created object 25... 2024-09-17T15:51:56.828 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39739]: created object 49... 2024-09-17T15:51:56.828 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39739]: finishing. 2024-09-17T15:51:56.828 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39739]: shutting down. 2024-09-17T15:51:56.832 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-17T15:51:56.833 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:51:56.838 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:56.838 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39741]: starting. 2024-09-17T15:51:56.838 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39741]: listing objects. 2024-09-17T15:51:56.838 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39741]: listed object 0... 2024-09-17T15:51:56.838 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39741]: listed object 25... 2024-09-17T15:51:56.838 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39741]: saw 50 objects 2024-09-17T15:51:56.838 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39741]: shutting down. 2024-09-17T15:51:56.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.932+0000 7f224d7c6640 10 monclient: get_monmap_and_config 2024-09-17T15:51:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.932+0000 7f224d7c6640 10 monclient: build_initial_monmap 2024-09-17T15:51:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.932+0000 7f224d7c6640 10 monclient: monmap: 2024-09-17T15:51:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:56.933024+0000 2024-09-17T15:51:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:56.933024+0000 2024-09-17T15:51:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:56.935 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:56.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.932+0000 7f224d7c6640 1 Processor -- start 2024-09-17T15:51:56.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.932+0000 7f224d7c6640 1 -- start start 2024-09-17T15:51:56.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.932+0000 7f224d7c6640 10 monclient: init 2024-09-17T15:51:56.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f224d7c6640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:56.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f224d7c6640 10 monclient: _add_conns ranks=[4,3,6,7,5,0,1,2,8] 2024-09-17T15:51:56.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f224d7c6640 10 monclient(hunting): picked mon.noname-c con 0x7f22480a54b0 addr v1:172.21.15.64:6791/0 2024-09-17T15:51:56.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f224d7c6640 10 monclient(hunting): picked mon.noname-g con 0x7f2248135c80 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:56.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f224d7c6640 10 monclient(hunting): picked mon.noname-d con 0x7f22481360a0 addr v1:172.21.15.64:6792/0 2024-09-17T15:51:56.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f224d7c6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22480aacb0 con 0x7f2248135c80 2024-09-17T15:51:56.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f224d7c6640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f224809d780 con 0x7f22480a54b0 2024-09-17T15:51:56.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f224d7c6640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f224809da60 con 0x7f22481360a0 2024-09-17T15:51:56.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f224d7c6640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:56.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f224d7c6640 10 monclient(hunting): authenticate will time out at 1114.608374s 2024-09-17T15:51:56.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2246ffd640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f22480a54b0 0x7f22480a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:57828/0 (socket says 172.21.15.64:57828) 2024-09-17T15:51:56.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f22477fe640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f22481360a0 0x7f2248141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:43828/0 (socket says 172.21.15.64:43828) 2024-09-17T15:51:56.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2246ffd640 1 -- 172.21.15.64:0/2419971878 learned_addr learned my addr 172.21.15.64:0/2419971878 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:56.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 1 -- 172.21.15.64:0/2419971878 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1592946597 0 0) 0x7f224809d780 con 0x7f22480a54b0 2024-09-17T15:51:56.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:56.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:56.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:56.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 10 monclient(hunting): my global_id is 154201 2024-09-17T15:51:56.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 1 -- 172.21.15.64:0/2419971878 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2234003520 con 0x7f22480a54b0 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 1 -- 172.21.15.64:0/2419971878 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3712638999 0 0) 0x7f224809da60 con 0x7f22481360a0 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 10 monclient(hunting): my global_id is 124782 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 1 -- 172.21.15.64:0/2419971878 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f224809d780 con 0x7f22481360a0 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 1 -- 172.21.15.64:0/2419971878 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 127416981 0 0) 0x7f22480aacb0 con 0x7f2248135c80 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 10 monclient(hunting): my global_id is 115527 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 1 -- 172.21.15.64:0/2419971878 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f224809da60 con 0x7f2248135c80 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.933+0000 7f2245ffb640 1 -- 172.21.15.64:0/2419971878 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1924420749 0 0) 0x7f2234003520 con 0x7f22480a54b0 2024-09-17T15:51:56.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 1 -- 172.21.15.64:0/2419971878 >> v1:172.21.15.64:6792/0 conn(0x7f22481360a0 legacy=0x7f2248141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 1 -- 172.21.15.64:0/2419971878 >> v1:172.21.15.100:6790/0 conn(0x7f2248135c80 legacy=0x7f224813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 1 monclient: found mon.noname-c 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 1 -- 172.21.15.64:0/2419971878 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f224809dd40 con 0x7f22480a54b0 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 10 monclient: _finish_auth 0 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 1 -- 172.21.15.64:0/2419971878 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2240001830 con 0x7f22480a54b0 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f224d7c6640 5 monclient: authenticate success, global_id 154201 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f224d7c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:56.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 10 monclient: dump: 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:56.947 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:56.948 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:56.948 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:56.948 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:56.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f224d7c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:56.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 1 -- 172.21.15.64:0/2419971878 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2240001a50 con 0x7f22480a54b0 2024-09-17T15:51:56.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:56.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 1 -- 172.21.15.64:0/2419971878 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2240004be0 con 0x7f22480a54b0 2024-09-17T15:51:56.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f224d7c6640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:56.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f224d7c6640 4 set_mon_vals no callback set 2024-09-17T15:51:56.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:56.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:56.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f2245ffb640 10 monclient: dump: 2024-09-17T15:51:56.948 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:56.949 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:56.950 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:56.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f224d7c6640 10 monclient: shutdown 2024-09-17T15:51:56.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f224d7c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:56.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f224d7c6640 1 -- 172.21.15.64:0/2419971878 >> v1:172.21.15.64:6791/0 conn(0x7f22480a54b0 legacy=0x7f22480a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:56.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f224d7c6640 1 -- 172.21.15.64:0/2419971878 shutdown_connections 2024-09-17T15:51:56.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f224d7c6640 1 -- 172.21.15.64:0/2419971878 >> 172.21.15.64:0/2419971878 conn(0x7f224806daa0 msgr2=0x7f22480ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:56.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f224d7c6640 1 -- 172.21.15.64:0/2419971878 shutdown_connections 2024-09-17T15:51:56.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.934+0000 7f224d7c6640 1 -- 172.21.15.64:0/2419971878 wait complete. 2024-09-17T15:51:56.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.935+0000 7f224d7c6640 10 monclient: build_initial_monmap 2024-09-17T15:51:56.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.935+0000 7f224d7c6640 10 monclient: monmap: 2024-09-17T15:51:56.950 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:56.950 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:56.950 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:56.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.935+0000 7f224d7c6640 1 Processor -- start 2024-09-17T15:51:56.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.935+0000 7f224d7c6640 1 -- start start 2024-09-17T15:51:56.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.935+0000 7f224d7c6640 10 monclient: init 2024-09-17T15:51:56.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f224d7c6640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:56.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f224d7c6640 10 monclient: _add_conns ranks=[5,1,2,7,0,4,6,8,3] 2024-09-17T15:51:56.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f224d7c6640 10 monclient(hunting): picked mon.noname-h con 0x7f22480a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:56.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f224d7c6640 10 monclient(hunting): picked mon.noname-f con 0x7f2248135c80 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:56.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f224d7c6640 10 monclient(hunting): picked mon.noname-b con 0x7f22481360a0 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:56.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f224d7c6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22480a2750 con 0x7f2248135c80 2024-09-17T15:51:56.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f224d7c6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22480a0170 con 0x7f22481360a0 2024-09-17T15:51:56.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f224d7c6640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22480a66b0 con 0x7f22480a54b0 2024-09-17T15:51:56.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f224d7c6640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:56.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f224d7c6640 10 monclient(hunting): authenticate will time out at 1114.611529s 2024-09-17T15:51:56.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f22477fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f22481360a0 0x7f22481d5e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51430/0 (socket says 172.21.15.64:51430) 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f22477fe640 1 -- 172.21.15.64:0/422537597 learned_addr learned my addr 172.21.15.64:0/422537597 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2281885828 0 0) 0x7f22480a0170 con 0x7f22481360a0 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 10 monclient(hunting): my global_id is 124868 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2228003520 con 0x7f22481360a0 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 148564496 0 0) 0x7f22480a2750 con 0x7f2248135c80 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 10 monclient(hunting): my global_id is 6598 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22480a0170 con 0x7f2248135c80 2024-09-17T15:51:56.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3567517732 0 0) 0x7f22480a66b0 con 0x7f22480a54b0 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 10 monclient(hunting): my global_id is 154220 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.936+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22480a2750 con 0x7f22480a54b0 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3564810137 0 0) 0x7f2228003520 con 0x7f22481360a0 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22480a66b0 con 0x7f22481360a0 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2238001d30 con 0x7f22481360a0 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient(hunting): dump: 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:56.954 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 347094912 0 0) 0x7f22480a0170 con 0x7f2248135c80 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2228003520 con 0x7f2248135c80 2024-09-17T15:51:56.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2230001550 con 0x7f2248135c80 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient(hunting): dump: 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:56.956 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3385921083 0 0) 0x7f22480a2750 con 0x7f22480a54b0 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22480a0170 con 0x7f22480a54b0 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1052294687 0 0) 0x7f22480a66b0 con 0x7f22481360a0 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 >> v1:172.21.15.100:6791/0 conn(0x7f22480a54b0 legacy=0x7f224809f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 >> v1:172.21.15.100:6789/0 conn(0x7f2248135c80 legacy=0x7f224809fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 monclient: found mon.b 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:56.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22480a2470 con 0x7f22481360a0 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f224d7c6640 5 monclient: authenticate success, global_id 124868 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f224d7c6640 10 monclient: _renew_subs 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f224d7c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f224d7c6640 1 -- 172.21.15.64:0/422537597 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22480a4fe0 con 0x7f22481360a0 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f224d7c6640 10 monclient: _renew_subs 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f224d7c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f224d7c6640 1 -- 172.21.15.64:0/422537597 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2248132aa0 con 0x7f22481360a0 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2238002050 con 0x7f22481360a0 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2238005150 con 0x7f22481360a0 2024-09-17T15:51:56.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f22457fa640 4 set_mon_vals no callback set 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.937+0000 7f223f7fe640 10 monclient: dump: 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:56.959 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:56.960 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.938+0000 7f224d7c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.938+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f2238011a30 con 0x7f22481360a0 2024-09-17T15:51:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.938+0000 7f224d7c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.938+0000 7f224d7c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:51:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.938+0000 7f224d7c6640 1 -- 172.21.15.64:0/422537597 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f222c005080 con 0x7f22481360a0 2024-09-17T15:51:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.938+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(287..287 src has 192..287) ==== 3898+0+0 (unknown 2029382806 0 0) 0x7f223804eeb0 con 0x7f22481360a0 2024-09-17T15:51:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.938+0000 7f2246ffd640 10 monclient: get_auth_request con 0x7f2228040c40 auth_method 0 2024-09-17T15:51:56.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.943+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f22380179e0 con 0x7f22481360a0 2024-09-17T15:51:56.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.943+0000 7f223f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:56.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:56.943+0000 7f223f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:57.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.204+0000 7f224d7c6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:51:57.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.204+0000 7f224d7c6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:57.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.204+0000 7f224d7c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:51:57.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.204+0000 7f224d7c6640 1 -- 172.21.15.64:0/422537597 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f222c005370 con 0x7f22481360a0 2024-09-17T15:51:57.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.205+0000 7f223f7fe640 1 -- 172.21.15.64:0/422537597 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 2639476209) 0x7f22380172e0 con 0x7f22481360a0 2024-09-17T15:51:57.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.205+0000 7f223f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:57.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.205+0000 7f223f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:57.207 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":32,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:51:57.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.207+0000 7f224d7c6640 1 -- 172.21.15.64:0/422537597 >> v1:172.21.15.100:6812/4210963484 conn(0x7f2228040c40 legacy=0x7f2228043100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.207+0000 7f224d7c6640 10 monclient: shutdown 2024-09-17T15:51:57.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.207+0000 7f224d7c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:57.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.207+0000 7f224d7c6640 1 -- 172.21.15.64:0/422537597 >> v1:172.21.15.64:6790/0 conn(0x7f22481360a0 legacy=0x7f22481d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.207+0000 7f224d7c6640 1 -- 172.21.15.64:0/422537597 shutdown_connections 2024-09-17T15:51:57.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.207+0000 7f224d7c6640 1 -- 172.21.15.64:0/422537597 >> 172.21.15.64:0/422537597 conn(0x7f224806daa0 msgr2=0x7f22480a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:57.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.207+0000 7f224d7c6640 1 -- 172.21.15.64:0/422537597 shutdown_connections 2024-09-17T15:51:57.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.207+0000 7f224d7c6640 1 -- 172.21.15.64:0/422537597 wait complete. 2024-09-17T15:51:57.221 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-17T15:51:57.221 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:51:57.221 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-17T15:51:57.221 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:51:57.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.320+0000 7f206e840640 10 monclient: get_monmap_and_config 2024-09-17T15:51:57.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.320+0000 7f206e840640 10 monclient: build_initial_monmap 2024-09-17T15:51:57.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.320+0000 7f206e840640 10 monclient: monmap: 2024-09-17T15:51:57.323 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:57.323 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.323 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:57.321784+0000 2024-09-17T15:51:57.323 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:57.321784+0000 2024-09-17T15:51:57.323 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:57.323 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.323 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:57.323 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:57.324 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:57.324 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:57.324 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:57.324 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:57.324 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:57.324 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:57.324 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:57.324 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.321+0000 7f206e840640 1 Processor -- start 2024-09-17T15:51:57.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.321+0000 7f206e840640 1 -- start start 2024-09-17T15:51:57.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.321+0000 7f206e840640 10 monclient: init 2024-09-17T15:51:57.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f206e840640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:57.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f206e840640 10 monclient: _add_conns ranks=[4,6,1,0,3,8,2,5,7] 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f206e840640 10 monclient(hunting): picked mon.noname-c con 0x7f20680a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f206e840640 10 monclient(hunting): picked mon.noname-d con 0x7f2068135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f206e840640 10 monclient(hunting): picked mon.noname-f con 0x7f2068135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f206e840640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f206809d6f0 con 0x7f2068135f70 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f206e840640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f206809d9d0 con 0x7f20680a5520 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f206e840640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f206809dcb0 con 0x7f2068135b50 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f206e840640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f206e840640 10 monclient(hunting): authenticate will time out at 1114.997394s 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f20677fe640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f2068135b50 0x7f206813deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:43832/0 (socket says 172.21.15.64:43832) 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f20677fe640 1 -- 172.21.15.64:0/3471115351 learned_addr learned my addr 172.21.15.64:0/3471115351 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2067fff640 1 --1- 172.21.15.64:0/3471115351 >> v1:172.21.15.64:6791/0 conn(0x7f20680a5520 0x7f20680a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:57842/0 (socket says 172.21.15.64:57842) 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 1 -- 172.21.15.64:0/3471115351 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1635993668 0 0) 0x7f206809d9d0 con 0x7f20680a5520 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 10 monclient(hunting): my global_id is 154210 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 1 -- 172.21.15.64:0/3471115351 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2050003520 con 0x7f20680a5520 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 1 -- 172.21.15.64:0/3471115351 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 645554098 0 0) 0x7f206809dcb0 con 0x7f2068135b50 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 10 monclient(hunting): my global_id is 124791 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 1 -- 172.21.15.64:0/3471115351 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f206809d9d0 con 0x7f2068135b50 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 1 -- 172.21.15.64:0/3471115351 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4255204757 0 0) 0x7f2050003520 con 0x7f20680a5520 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.322+0000 7f2066ffd640 1 -- 172.21.15.64:0/3471115351 >> v1:172.21.15.64:6792/0 conn(0x7f2068135b50 legacy=0x7f206813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 1 -- 172.21.15.64:0/3471115351 >> v1:172.21.15.100:6789/0 conn(0x7f2068135f70 legacy=0x7f20681415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:51:57.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 1 monclient: found mon.noname-c 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 1 -- 172.21.15.64:0/3471115351 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206809ccd0 con 0x7f20680a5520 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 10 monclient: _finish_auth 0 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 1 -- 172.21.15.64:0/3471115351 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2058001830 con 0x7f20680a5520 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 10 monclient: dump: 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.334 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f206e840640 5 monclient: authenticate success, global_id 154210 2024-09-17T15:51:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f206e840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:57.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 1 -- 172.21.15.64:0/3471115351 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2058001a50 con 0x7f20680a5520 2024-09-17T15:51:57.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:57.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 1 -- 172.21.15.64:0/3471115351 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2058001d40 con 0x7f20680a5520 2024-09-17T15:51:57.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f206e840640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:57.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f206e840640 4 set_mon_vals no callback set 2024-09-17T15:51:57.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:57.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f2066ffd640 10 monclient: dump: 2024-09-17T15:51:57.336 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.336 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.336 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.336 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.336 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.337 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.337 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.337 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.337 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.337 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.337 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.337 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.337 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.337 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.337 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.337 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f206e840640 10 monclient: shutdown 2024-09-17T15:51:57.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f206e840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f206e840640 1 -- 172.21.15.64:0/3471115351 >> v1:172.21.15.64:6791/0 conn(0x7f20680a5520 legacy=0x7f20680a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f206e840640 1 -- 172.21.15.64:0/3471115351 shutdown_connections 2024-09-17T15:51:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f206e840640 1 -- 172.21.15.64:0/3471115351 >> 172.21.15.64:0/3471115351 conn(0x7f206806db30 msgr2=0x7f20680ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f206e840640 1 -- 172.21.15.64:0/3471115351 shutdown_connections 2024-09-17T15:51:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.323+0000 7f206e840640 1 -- 172.21.15.64:0/3471115351 wait complete. 2024-09-17T15:51:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.324+0000 7f206e840640 10 monclient: build_initial_monmap 2024-09-17T15:51:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.324+0000 7f206e840640 10 monclient: monmap: 2024-09-17T15:51:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:57.338 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:57.339 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:57.339 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:57.339 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:57.339 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:57.339 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:57.339 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:57.339 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:57.339 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:57.339 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:57.339 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.324+0000 7f206e840640 1 Processor -- start 2024-09-17T15:51:57.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.324+0000 7f206e840640 1 -- start start 2024-09-17T15:51:57.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.324+0000 7f206e840640 10 monclient: init 2024-09-17T15:51:57.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f206e840640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:57.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f206e840640 10 monclient: _add_conns ranks=[0,7,5,2,4,3,6,1,8] 2024-09-17T15:51:57.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f206e840640 10 monclient(hunting): picked mon.noname-a con 0x7f20680a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:57.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f206e840640 10 monclient(hunting): picked mon.noname-i con 0x7f2068135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:57.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f206e840640 10 monclient(hunting): picked mon.noname-h con 0x7f2068135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:57.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f206e840640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20680a0620 con 0x7f20680a5520 2024-09-17T15:51:57.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f206e840640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2068134f30 con 0x7f2068135f70 2024-09-17T15:51:57.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f206e840640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20680a9280 con 0x7f2068135b50 2024-09-17T15:51:57.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f206e840640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:57.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f206e840640 10 monclient(hunting): authenticate will time out at 1115.000346s 2024-09-17T15:51:57.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2067fff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f20680a5520 0x7f20680a4b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:42608/0 (socket says 172.21.15.64:42608) 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2067fff640 1 -- 172.21.15.64:0/1315178328 learned_addr learned my addr 172.21.15.64:0/1315178328 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2387955043 0 0) 0x7f20680a0620 con 0x7f20680a5520 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): my global_id is 124803 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2054003520 con 0x7f20680a5520 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3687220596 0 0) 0x7f2054003520 con 0x7f20680a5520 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20680a0620 con 0x7f20680a5520 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f20580017f0 con 0x7f20680a5520 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): dump: 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.341 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:57.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3231100259 0 0) 0x7f2068134f30 con 0x7f2068135f70 2024-09-17T15:51:57.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): my global_id is 154229 2024-09-17T15:51:57.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2054003520 con 0x7f2068135f70 2024-09-17T15:51:57.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3386748393 0 0) 0x7f20680a9280 con 0x7f2068135b50 2024-09-17T15:51:57.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 10 monclient(hunting): my global_id is 154240 2024-09-17T15:51:57.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.325+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2068134f30 con 0x7f2068135b50 2024-09-17T15:51:57.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3943290377 0 0) 0x7f20680a0620 con 0x7f20680a5520 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 >> v1:172.21.15.100:6792/0 conn(0x7f2068135b50 legacy=0x7f20681d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 >> v1:172.21.15.100:6791/0 conn(0x7f2068135f70 legacy=0x7f20681d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 10 monclient: _finish_hunting 0 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 1 monclient: found mon.a 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20680a8d30 con 0x7f20680a5520 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 10 monclient: _finish_auth 0 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f206e840640 5 monclient: authenticate success, global_id 124803 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f206e840640 10 monclient: _renew_subs 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f206e840640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:51:57.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f206e840640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f20680a0340 con 0x7f20680a5520 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f206e840640 10 monclient: _renew_subs 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f206e840640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f206e840640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f20680a5270 con 0x7f20680a5520 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2058004f30 con 0x7f20680a5520 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2058005420 con 0x7f20680a5520 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f20667fc640 4 set_mon_vals no callback set 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f2064ff9640 10 monclient: dump: 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.345 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f206e840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.326+0000 7f206e840640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f20480065c0 con 0x7f2048002860 2024-09-17T15:51:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f2058011d00 con 0x7f20680a5520 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(287..287 src has 192..287) ==== 3898+0+0 (unknown 2029382806 0 0) 0x7f205804ecc0 con 0x7f20680a5520 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3877881849 0 0) 0x7f20480065c0 con 0x7f2048002860 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 10 monclient(hunting): my global_id is 124812 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20680a0620 con 0x7f2048002860 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f206cddd640 10 monclient: get_auth_request con 0x7f2054040420 auth_method 0 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1949634043 0 0) 0x7f20680a0620 con 0x7f2048002860 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20480065c0 con 0x7f2048002860 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f205c001470 con 0x7f2048002860 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:57.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 10 monclient: dump: 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.348 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.349 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.349 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 332713645 0 0) 0x7f20480065c0 con 0x7f2048002860 2024-09-17T15:51:57.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.327+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.64:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f204800a160 con 0x7f2048002860 2024-09-17T15:51:57.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.328+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f205c0017c0 con 0x7f2048002860 2024-09-17T15:51:57.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.328+0000 7f2064ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:57.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.328+0000 7f2064ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:57.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.328+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 >> v1:172.21.15.64:6789/0 conn(0x7f2048002860 legacy=0x7f2048004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.328+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 >> v1:172.21.15.64:6789/0 conn(0x7f2048002860 legacy=0x7f2048004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:57.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f206e840640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:57.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f206e840640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2048009e70 con 0x7f2048009130 2024-09-17T15:51:57.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3465397407 0 0) 0x7f2048009e70 con 0x7f2048009130 2024-09-17T15:51:57.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 10 monclient(hunting): my global_id is 124821 2024-09-17T15:51:57.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20480065c0 con 0x7f2048009130 2024-09-17T15:51:57.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1505493134 0 0) 0x7f20480065c0 con 0x7f2048009130 2024-09-17T15:51:57.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2048009e70 con 0x7f2048009130 2024-09-17T15:51:57.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f205c004d10 con 0x7f2048009130 2024-09-17T15:51:57.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:57.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 10 monclient: dump: 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.352 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.353 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.353 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3459596411 0 0) 0x7f2048009e70 con 0x7f2048009130 2024-09-17T15:51:57.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.334+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 --> v1:172.21.15.64:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f204800c410 con 0x7f2048009130 2024-09-17T15:51:57.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3896575143) 0x7f205c006280 con 0x7f2048009130 2024-09-17T15:51:57.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f2064ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:57.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f2064ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:57.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 >> v1:172.21.15.64:6789/0 conn(0x7f2048009130 legacy=0x7f20480095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f2064ff9640 1 -- 172.21.15.64:0/1315178328 >> v1:172.21.15.64:6789/0 conn(0x7f2048009130 legacy=0x7f20480095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:57.353 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 124253, 2024-09-17T15:51:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 3, 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 4 2024-09-17T15:51:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:57.372 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:57.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f206e840640 1 -- 172.21.15.64:0/1315178328 >> v1:172.21.15.100:6812/4210963484 conn(0x7f2054040420 legacy=0x7f20540428e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f206e840640 10 monclient: shutdown 2024-09-17T15:51:57.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f206e840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:57.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f206e840640 1 -- 172.21.15.64:0/1315178328 >> v1:172.21.15.64:6789/0 conn(0x7f20680a5520 legacy=0x7f20680a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f206cddd640 1 -- 172.21.15.64:0/1315178328 reap_dead start 2024-09-17T15:51:57.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f206cddd640 1 -- 172.21.15.64:0/1315178328 reap_dead start 2024-09-17T15:51:57.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f206e840640 1 -- 172.21.15.64:0/1315178328 shutdown_connections 2024-09-17T15:51:57.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f206e840640 1 -- 172.21.15.64:0/1315178328 >> 172.21.15.64:0/1315178328 conn(0x7f206806db30 msgr2=0x7f206812c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:57.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f206e840640 1 -- 172.21.15.64:0/1315178328 shutdown_connections 2024-09-17T15:51:57.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.335+0000 7f206e840640 1 -- 172.21.15.64:0/1315178328 wait complete. 2024-09-17T15:51:57.374 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:57.375 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-17T15:51:57.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:51:57.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.474+0000 7f089bd52640 10 monclient: get_monmap_and_config 2024-09-17T15:51:57.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.474+0000 7f089bd52640 10 monclient: build_initial_monmap 2024-09-17T15:51:57.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.474+0000 7f089bd52640 10 monclient: monmap: 2024-09-17T15:51:57.477 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:57.477 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.477 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:57.475755+0000 2024-09-17T15:51:57.477 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:57.475755+0000 2024-09-17T15:51:57.477 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:57.477 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.477 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:57.477 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:57.478 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:57.478 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:57.478 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:57.478 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:57.478 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:57.478 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:57.478 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:57.478 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.475+0000 7f089bd52640 1 Processor -- start 2024-09-17T15:51:57.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.475+0000 7f089bd52640 1 -- start start 2024-09-17T15:51:57.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.475+0000 7f089bd52640 10 monclient: init 2024-09-17T15:51:57.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f089bd52640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:57.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f089bd52640 10 monclient: _add_conns ranks=[2,4,3,7,6,0,1,8,5] 2024-09-17T15:51:57.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f089bd52640 10 monclient(hunting): picked mon.noname-b con 0x7f08940a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:57.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f089bd52640 10 monclient(hunting): picked mon.noname-c con 0x7f0894135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:51:57.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f089bd52640 10 monclient(hunting): picked mon.noname-g con 0x7f0894135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:57.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f089bd52640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f089409d6f0 con 0x7f08940a5520 2024-09-17T15:51:57.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f089bd52640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f089409d9d0 con 0x7f0894135f70 2024-09-17T15:51:57.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f089bd52640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f089409dcb0 con 0x7f0894135b50 2024-09-17T15:51:57.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f089bd52640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:57.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f089bd52640 10 monclient(hunting): authenticate will time out at 1115.151256s 2024-09-17T15:51:57.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f08992ed640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f0894135b50 0x7f089413deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:57846/0 (socket says 172.21.15.64:57846) 2024-09-17T15:51:57.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f08992ed640 1 -- 172.21.15.64:0/195081770 learned_addr learned my addr 172.21.15.64:0/195081770 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:57.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0899aee640 1 --1- 172.21.15.64:0/195081770 >> v1:172.21.15.64:6790/0 conn(0x7f08940a5520 0x7f08940a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51432/0 (socket says 172.21.15.64:51432) 2024-09-17T15:51:57.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 1 -- 172.21.15.64:0/195081770 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 53278001 0 0) 0x7f089409dcb0 con 0x7f0894135b50 2024-09-17T15:51:57.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient(hunting): my global_id is 154219 2024-09-17T15:51:57.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 1 -- 172.21.15.64:0/195081770 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f087c003520 con 0x7f0894135b50 2024-09-17T15:51:57.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 1 -- 172.21.15.64:0/195081770 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 826287325 0 0) 0x7f089409d6f0 con 0x7f08940a5520 2024-09-17T15:51:57.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient(hunting): my global_id is 124877 2024-09-17T15:51:57.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 1 -- 172.21.15.64:0/195081770 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f089409dcb0 con 0x7f08940a5520 2024-09-17T15:51:57.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 1 -- 172.21.15.64:0/195081770 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3157735511 0 0) 0x7f089409d9d0 con 0x7f0894135f70 2024-09-17T15:51:57.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient(hunting): my global_id is 115536 2024-09-17T15:51:57.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 1 -- 172.21.15.64:0/195081770 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f089409d6f0 con 0x7f0894135f70 2024-09-17T15:51:57.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 1 -- 172.21.15.64:0/195081770 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2783786295 0 0) 0x7f087c003520 con 0x7f0894135b50 2024-09-17T15:51:57.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 1 -- 172.21.15.64:0/195081770 >> v1:172.21.15.100:6790/0 conn(0x7f0894135f70 legacy=0x7f08941415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 1 -- 172.21.15.64:0/195081770 >> v1:172.21.15.64:6790/0 conn(0x7f08940a5520 legacy=0x7f08940a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient: _finish_hunting 0 2024-09-17T15:51:57.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 1 monclient: found mon.noname-c 2024-09-17T15:51:57.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:57.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:51:57.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 1 -- 172.21.15.64:0/195081770 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089409ccd0 con 0x7f0894135b50 2024-09-17T15:51:57.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient: _finish_auth 0 2024-09-17T15:51:57.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient: _check_auth_tickets 2024-09-17T15:51:57.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:57.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f089bd52640 5 monclient: authenticate success, global_id 154219 2024-09-17T15:51:57.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 1 -- 172.21.15.64:0/195081770 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0888001830 con 0x7f0894135b50 2024-09-17T15:51:57.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f089bd52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:57.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:57.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.476+0000 7f0898aec640 10 monclient: dump: 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.490 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.491 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.491 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f089bd52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:57.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f0898aec640 1 -- 172.21.15.64:0/195081770 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f08880019d0 con 0x7f0894135b50 2024-09-17T15:51:57.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f0898aec640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:57.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f089bd52640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:57.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f0898aec640 1 -- 172.21.15.64:0/195081770 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0888004be0 con 0x7f0894135b50 2024-09-17T15:51:57.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f089bd52640 4 set_mon_vals no callback set 2024-09-17T15:51:57.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f0898aec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f0898aec640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:57.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f0898aec640 10 monclient: dump: 2024-09-17T15:51:57.491 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.491 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.492 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.492 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.492 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.492 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.492 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.492 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.492 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.492 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.492 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.492 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.492 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.492 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.492 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.493 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f089bd52640 10 monclient: shutdown 2024-09-17T15:51:57.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f089bd52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:57.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f089bd52640 1 -- 172.21.15.64:0/195081770 >> v1:172.21.15.64:6791/0 conn(0x7f0894135b50 legacy=0x7f089413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f089bd52640 1 -- 172.21.15.64:0/195081770 shutdown_connections 2024-09-17T15:51:57.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f089bd52640 1 -- 172.21.15.64:0/195081770 >> 172.21.15.64:0/195081770 conn(0x7f089406db30 msgr2=0x7f08940ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:57.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f089bd52640 1 -- 172.21.15.64:0/195081770 shutdown_connections 2024-09-17T15:51:57.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.477+0000 7f089bd52640 1 -- 172.21.15.64:0/195081770 wait complete. 2024-09-17T15:51:57.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.478+0000 7f089bd52640 10 monclient: build_initial_monmap 2024-09-17T15:51:57.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.478+0000 7f089bd52640 10 monclient: monmap: 2024-09-17T15:51:57.493 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:57.493 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:57.493 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.478+0000 7f089bd52640 1 Processor -- start 2024-09-17T15:51:57.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.478+0000 7f089bd52640 1 -- start start 2024-09-17T15:51:57.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.478+0000 7f089bd52640 10 monclient: init 2024-09-17T15:51:57.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:57.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 10 monclient: _add_conns ranks=[4,5,3,2,7,1,8,0,6] 2024-09-17T15:51:57.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 10 monclient(hunting): picked mon.noname-c con 0x7f08940a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:51:57.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 10 monclient(hunting): picked mon.noname-h con 0x7f0894135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:57.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 10 monclient(hunting): picked mon.noname-g con 0x7f0894135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:57.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08940a44c0 con 0x7f0894135f70 2024-09-17T15:51:57.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08940a02a0 con 0x7f08940a5520 2024-09-17T15:51:57.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08940a0580 con 0x7f0894135b50 2024-09-17T15:51:57.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:57.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 10 monclient(hunting): authenticate will time out at 1115.154133s 2024-09-17T15:51:57.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f0899aee640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f08940a5520 0x7f0894131a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:57858/0 (socket says 172.21.15.64:57858) 2024-09-17T15:51:57.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f0899aee640 1 -- 172.21.15.64:0/2447291599 learned_addr learned my addr 172.21.15.64:0/2447291599 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:57.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1038977342 0 0) 0x7f08940a02a0 con 0x7f08940a5520 2024-09-17T15:51:57.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): my global_id is 154228 2024-09-17T15:51:57.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0880003520 con 0x7f08940a5520 2024-09-17T15:51:57.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1896861603 0 0) 0x7f0880003520 con 0x7f08940a5520 2024-09-17T15:51:57.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08940a02a0 con 0x7f08940a5520 2024-09-17T15:51:57.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f088c0015b0 con 0x7f08940a5520 2024-09-17T15:51:57.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:57.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:57.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): dump: 2024-09-17T15:51:57.497 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.497 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.497 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.497 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.497 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.497 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.497 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.497 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.497 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.497 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.497 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.497 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.497 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 907979264 0 0) 0x7f08940a44c0 con 0x7f0894135f70 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): my global_id is 115545 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0880003520 con 0x7f0894135f70 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2136147930 0 0) 0x7f08940a0580 con 0x7f0894135b50 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient(hunting): my global_id is 154238 2024-09-17T15:51:57.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08940a44c0 con 0x7f0894135b50 2024-09-17T15:51:57.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 702939318 0 0) 0x7f08940a02a0 con 0x7f08940a5520 2024-09-17T15:51:57.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 >> v1:172.21.15.100:6791/0 conn(0x7f0894135b50 legacy=0x7f0894132150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 >> v1:172.21.15.100:6790/0 conn(0x7f0894135f70 legacy=0x7f08941d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient: _finish_hunting 0 2024-09-17T15:51:57.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 monclient: found mon.c 2024-09-17T15:51:57.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:57.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:57.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08940a8e50 con 0x7f08940a5520 2024-09-17T15:51:57.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient: _finish_auth 0 2024-09-17T15:51:57.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient: _check_auth_tickets 2024-09-17T15:51:57.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:57.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 5 monclient: authenticate success, global_id 154228 2024-09-17T15:51:57.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 10 monclient: _renew_subs 2024-09-17T15:51:57.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:57.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f089409f9f0 con 0x7f08940a5520 2024-09-17T15:51:57.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 10 monclient: _renew_subs 2024-09-17T15:51:57.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:57.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f089bd52640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f089409e550 con 0x7f08940a5520 2024-09-17T15:51:57.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f088c004a90 con 0x7f08940a5520 2024-09-17T15:51:57.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:57.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f088c004f80 con 0x7f08940a5520 2024-09-17T15:51:57.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f08927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:57.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.480+0000 7f08927fc640 10 monclient: dump: 2024-09-17T15:51:57.501 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.501 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.501 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.501 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.501 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.501 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.501 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.501 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.501 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.501 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.502 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.502 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.502 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.502 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.502 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.502 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.479+0000 7f0893fff640 4 set_mon_vals no callback set 2024-09-17T15:51:57.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.480+0000 7f089bd52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:57.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.480+0000 7f089bd52640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f08940a7b20 con 0x7f0894142f10 2024-09-17T15:51:57.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.480+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f088c0118e0 con 0x7f08940a5520 2024-09-17T15:51:57.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(287..287 src has 192..287) ==== 3898+0+0 (unknown 2029382806 0 0) 0x7f088c04ec40 con 0x7f08940a5520 2024-09-17T15:51:57.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 741547334 0 0) 0x7f08940a44c0 con 0x7f0894142f10 2024-09-17T15:51:57.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 10 monclient(hunting): my global_id is 124886 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08940a02a0 con 0x7f0894142f10 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f089a2ef640 10 monclient: get_auth_request con 0x7f08800404d0 auth_method 0 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1940310222 0 0) 0x7f08940a02a0 con 0x7f0894142f10 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08940a44c0 con 0x7f0894142f10 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0888004c20 con 0x7f0894142f10 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 10 monclient: dump: 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.503 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.504 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.504 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.504 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.504 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.504 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.504 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.504 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.504 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.504 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.504 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.504 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.504 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.504 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2494001766 0 0) 0x7f08940a44c0 con 0x7f0894142f10 2024-09-17T15:51:57.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.64:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f08940a50f0 con 0x7f0894142f10 2024-09-17T15:51:57.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0888001550 con 0x7f0894142f10 2024-09-17T15:51:57.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:57.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.481+0000 7f08927fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:57.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.482+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 >> v1:172.21.15.64:6790/0 conn(0x7f0894142f10 legacy=0x7f089412d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.482+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 >> v1:172.21.15.64:6790/0 conn(0x7f0894142f10 legacy=0x7f089412d6f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:57.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f089bd52640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:57.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f089bd52640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0894068b50 con 0x7f089402e450 2024-09-17T15:51:57.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 628125544 0 0) 0x7f0894068b50 con 0x7f089402e450 2024-09-17T15:51:57.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 10 monclient(hunting): my global_id is 124895 2024-09-17T15:51:57.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08940a44c0 con 0x7f089402e450 2024-09-17T15:51:57.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2439741861 0 0) 0x7f08940a44c0 con 0x7f089402e450 2024-09-17T15:51:57.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0894068b50 con 0x7f089402e450 2024-09-17T15:51:57.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0888005000 con 0x7f089402e450 2024-09-17T15:51:57.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:57.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 10 monclient: dump: 2024-09-17T15:51:57.508 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.508 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.508 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.508 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.508 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.508 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.508 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.508 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.508 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.508 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.509 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.509 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.509 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.509 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.509 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.509 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3054814950 0 0) 0x7f0894068b50 con 0x7f089402e450 2024-09-17T15:51:57.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.488+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 --> v1:172.21.15.64:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f089402e830 con 0x7f089402e450 2024-09-17T15:51:57.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3295001073) 0x7f0888005130 con 0x7f089402e450 2024-09-17T15:51:57.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f08927fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:57.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f08927fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:57.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 >> v1:172.21.15.64:6790/0 conn(0x7f089402e450 legacy=0x7f089412de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f08927fc640 1 -- 172.21.15.64:0/2447291599 >> v1:172.21.15.64:6790/0 conn(0x7f089402e450 legacy=0x7f089412de00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:57.510 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 124406, 2024-09-17T15:51:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 3, 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.530 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:57.530 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:57.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f089bd52640 1 -- 172.21.15.64:0/2447291599 >> v1:172.21.15.100:6812/4210963484 conn(0x7f08800404d0 legacy=0x7f0880042990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f089bd52640 10 monclient: shutdown 2024-09-17T15:51:57.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f089bd52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:57.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f089bd52640 1 -- 172.21.15.64:0/2447291599 >> v1:172.21.15.64:6791/0 conn(0x7f08940a5520 legacy=0x7f0894131a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f089a2ef640 1 -- 172.21.15.64:0/2447291599 reap_dead start 2024-09-17T15:51:57.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f089a2ef640 1 -- 172.21.15.64:0/2447291599 reap_dead start 2024-09-17T15:51:57.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f089bd52640 1 -- 172.21.15.64:0/2447291599 shutdown_connections 2024-09-17T15:51:57.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f089bd52640 1 -- 172.21.15.64:0/2447291599 >> 172.21.15.64:0/2447291599 conn(0x7f089406db30 msgr2=0x7f089412c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:57.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f089bd52640 1 -- 172.21.15.64:0/2447291599 shutdown_connections 2024-09-17T15:51:57.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.489+0000 7f089bd52640 1 -- 172.21.15.64:0/2447291599 wait complete. 2024-09-17T15:51:57.532 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:57.532 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-17T15:51:57.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:51:57.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.632+0000 7f2ea5e72640 10 monclient: get_monmap_and_config 2024-09-17T15:51:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.632+0000 7f2ea5e72640 10 monclient: build_initial_monmap 2024-09-17T15:51:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.632+0000 7f2ea5e72640 10 monclient: monmap: 2024-09-17T15:51:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:57.633090+0000 2024-09-17T15:51:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:57.633090+0000 2024-09-17T15:51:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:57.635 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.632+0000 7f2ea5e72640 1 Processor -- start 2024-09-17T15:51:57.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2ea5e72640 1 -- start start 2024-09-17T15:51:57.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2ea5e72640 10 monclient: init 2024-09-17T15:51:57.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2ea5e72640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:57.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2ea5e72640 10 monclient: _add_conns ranks=[1,5,8,4,6,7,0,2,3] 2024-09-17T15:51:57.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2ea5e72640 10 monclient(hunting): picked mon.noname-f con 0x7f2ea00a54b0 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:57.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2ea5e72640 10 monclient(hunting): picked mon.noname-h con 0x7f2ea0135c80 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:57.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2ea5e72640 10 monclient(hunting): picked mon.noname-e con 0x7f2ea01360a0 addr v1:172.21.15.64:6793/0 2024-09-17T15:51:57.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2ea5e72640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea00aacb0 con 0x7f2ea00a54b0 2024-09-17T15:51:57.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2ea5e72640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea009d780 con 0x7f2ea0135c80 2024-09-17T15:51:57.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2ea5e72640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea009da60 con 0x7f2ea01360a0 2024-09-17T15:51:57.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2ea5e72640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:57.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2ea5e72640 10 monclient(hunting): authenticate will time out at 1115.308581s 2024-09-17T15:51:57.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2e9ffff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f2ea01360a0 0x7f2ea0141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:37074/0 (socket says 172.21.15.64:37074) 2024-09-17T15:51:57.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2e9ffff640 1 -- 172.21.15.64:0/1256041645 learned_addr learned my addr 172.21.15.64:0/1256041645 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:57.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/1256041645 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1517685132 0 0) 0x7f2ea009da60 con 0x7f2ea01360a0 2024-09-17T15:51:57.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2e9e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2e9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2e9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.633+0000 7f2e9e7fc640 10 monclient(hunting): my global_id is 154178 2024-09-17T15:51:57.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/1256041645 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e8c003520 con 0x7f2ea01360a0 2024-09-17T15:51:57.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/1256041645 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1311619659 0 0) 0x7f2ea009d780 con 0x7f2ea0135c80 2024-09-17T15:51:57.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient(hunting): my global_id is 154247 2024-09-17T15:51:57.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/1256041645 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ea009da60 con 0x7f2ea0135c80 2024-09-17T15:51:57.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/1256041645 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2829526052 0 0) 0x7f2ea00aacb0 con 0x7f2ea00a54b0 2024-09-17T15:51:57.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient(hunting): my global_id is 6616 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/1256041645 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ea009d780 con 0x7f2ea00a54b0 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/1256041645 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1603906209 0 0) 0x7f2e8c003520 con 0x7f2ea01360a0 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/1256041645 >> v1:172.21.15.100:6791/0 conn(0x7f2ea0135c80 legacy=0x7f2ea013df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/1256041645 >> v1:172.21.15.100:6789/0 conn(0x7f2ea00a54b0 legacy=0x7f2ea00a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 1 monclient: found mon.noname-e 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/1256041645 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea009dd40 con 0x7f2ea01360a0 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:51:57.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/1256041645 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e98001bb0 con 0x7f2ea01360a0 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2ea5e72640 5 monclient: authenticate success, global_id 154178 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2ea5e72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient: dump: 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.647 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2ea5e72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/1256041645 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2e98001d50 con 0x7f2ea01360a0 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 1 -- 172.21.15.64:0/1256041645 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e98004f60 con 0x7f2ea01360a0 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:51:57.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2e9e7fc640 10 monclient: dump: 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.649 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.650 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.650 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2ea5e72640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:57.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2ea5e72640 4 set_mon_vals no callback set 2024-09-17T15:51:57.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2ea5e72640 10 monclient: shutdown 2024-09-17T15:51:57.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2ea5e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:57.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2ea5e72640 1 -- 172.21.15.64:0/1256041645 >> v1:172.21.15.64:6793/0 conn(0x7f2ea01360a0 legacy=0x7f2ea0141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2ea5e72640 1 -- 172.21.15.64:0/1256041645 shutdown_connections 2024-09-17T15:51:57.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2ea5e72640 1 -- 172.21.15.64:0/1256041645 >> 172.21.15.64:0/1256041645 conn(0x7f2ea006daa0 msgr2=0x7f2ea00ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:57.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2ea5e72640 1 -- 172.21.15.64:0/1256041645 shutdown_connections 2024-09-17T15:51:57.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.634+0000 7f2ea5e72640 1 -- 172.21.15.64:0/1256041645 wait complete. 2024-09-17T15:51:57.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.635+0000 7f2ea5e72640 10 monclient: build_initial_monmap 2024-09-17T15:51:57.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.635+0000 7f2ea5e72640 10 monclient: monmap: 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:57.651 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.635+0000 7f2ea5e72640 1 Processor -- start 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2ea5e72640 1 -- start start 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2ea5e72640 10 monclient: init 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2ea5e72640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2ea5e72640 10 monclient: _add_conns ranks=[5,8,2,4,7,3,6,1,0] 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2ea5e72640 10 monclient(hunting): picked mon.noname-h con 0x7f2ea00a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2ea5e72640 10 monclient(hunting): picked mon.noname-e con 0x7f2ea0135c80 addr v1:172.21.15.64:6793/0 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2ea5e72640 10 monclient(hunting): picked mon.noname-b con 0x7f2ea01360a0 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2ea5e72640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea009eae0 con 0x7f2ea01360a0 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2ea5e72640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea00a6fd0 con 0x7f2ea00a54b0 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2ea5e72640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea01352a0 con 0x7f2ea0135c80 2024-09-17T15:51:57.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2ea5e72640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:57.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2ea5e72640 10 monclient(hunting): authenticate will time out at 1115.311636s 2024-09-17T15:51:57.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e9ffff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f2ea01360a0 0x7f2ea01d5e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51462/0 (socket says 172.21.15.64:51462) 2024-09-17T15:51:57.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e9effd640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f2ea0135c80 0x7f2ea0131fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:37086/0 (socket says 172.21.15.64:37086) 2024-09-17T15:51:57.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e9ffff640 1 -- 172.21.15.64:0/2324818425 learned_addr learned my addr 172.21.15.64:0/2324818425 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:57.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2430312160 0 0) 0x7f2e88001130 con 0x7f2ea0135c80 2024-09-17T15:51:57.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e97fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e97fff640 10 monclient(hunting): my global_id is 154187 2024-09-17T15:51:57.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e80003520 con 0x7f2ea0135c80 2024-09-17T15:51:57.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 146045762 0 0) 0x7f2ea009eae0 con 0x7f2ea01360a0 2024-09-17T15:51:57.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e97fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.636+0000 7f2e97fff640 10 monclient(hunting): my global_id is 124904 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e88001130 con 0x7f2ea01360a0 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3875937966 0 0) 0x7f2ea009d780 con 0x7f2ea00a54b0 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): my global_id is 154256 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ea009eae0 con 0x7f2ea00a54b0 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1241888247 0 0) 0x7f2e80003520 con 0x7f2ea0135c80 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ea009d780 con 0x7f2ea0135c80 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e880035f0 con 0x7f2ea0135c80 2024-09-17T15:51:57.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:57.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:51:57.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): dump: 2024-09-17T15:51:57.655 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.655 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.655 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.655 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.655 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.655 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.655 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.655 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.655 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.655 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.655 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1595319071 0 0) 0x7f2e88001130 con 0x7f2ea01360a0 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e80003520 con 0x7f2ea01360a0 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 244342607 0 0) 0x7f2ea009eae0 con 0x7f2ea00a54b0 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e88001130 con 0x7f2ea00a54b0 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e98001b70 con 0x7f2ea01360a0 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:57.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): dump: 2024-09-17T15:51:57.657 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.657 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.657 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.657 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.657 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.657 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.657 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.657 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.657 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.657 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.657 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.657 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.657 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.658 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.658 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.658 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e90001a10 con 0x7f2ea00a54b0 2024-09-17T15:51:57.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:57.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:51:57.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient(hunting): dump: 2024-09-17T15:51:57.658 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.658 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.658 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.658 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.658 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.658 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2072006648 0 0) 0x7f2ea009d780 con 0x7f2ea0135c80 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 >> v1:172.21.15.100:6791/0 conn(0x7f2ea00a54b0 legacy=0x7f2ea01318a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 >> v1:172.21.15.64:6790/0 conn(0x7f2ea01360a0 legacy=0x7f2ea01d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:57.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 monclient: found mon.e 2024-09-17T15:51:57.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:57.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:57.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea00a0130 con 0x7f2ea0135c80 2024-09-17T15:51:57.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient: _finish_auth 0 2024-09-17T15:51:57.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient: _check_auth_tickets 2024-09-17T15:51:57.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:57.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2ea5e72640 5 monclient: authenticate success, global_id 154187 2024-09-17T15:51:57.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2ea5e72640 10 monclient: _renew_subs 2024-09-17T15:51:57.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2ea5e72640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:57.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2ea5e72640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea00a0410 con 0x7f2ea0135c80 2024-09-17T15:51:57.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2ea5e72640 10 monclient: _renew_subs 2024-09-17T15:51:57.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2ea5e72640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:57.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2ea5e72640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f2ea00a42e0 con 0x7f2ea0135c80 2024-09-17T15:51:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2e88003a40 con 0x7f2ea0135c80 2024-09-17T15:51:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e88003f70 con 0x7f2ea0135c80 2024-09-17T15:51:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:51:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e9dffb640 4 set_mon_vals no callback set 2024-09-17T15:51:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.637+0000 7f2e97fff640 10 monclient: dump: 2024-09-17T15:51:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.638+0000 7f2ea5e72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.638+0000 7f2ea5e72640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2e840065c0 con 0x7f2e84002860 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.638+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f2e88004110 con 0x7f2ea0135c80 2024-09-17T15:51:57.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.638+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(287..287 src has 192..287) ==== 3898+0+0 (unknown 2029382806 0 0) 0x7f2e8804ebc0 con 0x7f2ea0135c80 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.638+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1756170596 0 0) 0x7f2e840065c0 con 0x7f2e84002860 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.638+0000 7f2e97fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.638+0000 7f2e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.638+0000 7f2e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.638+0000 7f2e97fff640 10 monclient(hunting): my global_id is 154237 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.638+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ea009d780 con 0x7f2e84002860 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e9ffff640 10 monclient: get_auth_request con 0x7f2e80040d00 auth_method 0 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1170255737 0 0) 0x7f2ea009d780 con 0x7f2e84002860 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e840065c0 con 0x7f2e84002860 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e90003930 con 0x7f2e84002860 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e97fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e97fff640 10 monclient: dump: 2024-09-17T15:51:57.663 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.664 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.665 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3356110583 0 0) 0x7f2e840065c0 con 0x7f2e84002860 2024-09-17T15:51:57.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e8400a160 con 0x7f2e84002860 2024-09-17T15:51:57.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2e90003b60 con 0x7f2e84002860 2024-09-17T15:51:57.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e97fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:57.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e97fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:57.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 >> v1:172.21.15.64:6791/0 conn(0x7f2e84002860 legacy=0x7f2e84004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.639+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 >> v1:172.21.15.64:6791/0 conn(0x7f2e84002860 legacy=0x7f2e84004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:57.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.645+0000 7f2ea5e72640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:57.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.645+0000 7f2ea5e72640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2e84009e20 con 0x7f2e840090e0 2024-09-17T15:51:57.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.645+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2239165218 0 0) 0x7f2e84009e20 con 0x7f2e840090e0 2024-09-17T15:51:57.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.645+0000 7f2e97fff640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.645+0000 7f2e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.645+0000 7f2e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.645+0000 7f2e97fff640 10 monclient(hunting): my global_id is 154246 2024-09-17T15:51:57.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.645+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e840065c0 con 0x7f2e840090e0 2024-09-17T15:51:57.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3881695520 0 0) 0x7f2e840065c0 con 0x7f2e840090e0 2024-09-17T15:51:57.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e84009e20 con 0x7f2e840090e0 2024-09-17T15:51:57.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2e90004340 con 0x7f2e840090e0 2024-09-17T15:51:57.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e97fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e97fff640 10 monclient: dump: 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.668 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.669 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.669 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.669 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.669 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2644676468 0 0) 0x7f2e84009e20 con 0x7f2e840090e0 2024-09-17T15:51:57.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 --> v1:172.21.15.64:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2e8400c3f0 con 0x7f2e840090e0 2024-09-17T15:51:57.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 1698350593) 0x7f2e90006600 con 0x7f2e840090e0 2024-09-17T15:51:57.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e97fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:57.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e97fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:57.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 >> v1:172.21.15.64:6791/0 conn(0x7f2e840090e0 legacy=0x7f2e84009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e97fff640 1 -- 172.21.15.64:0/2324818425 >> v1:172.21.15.64:6791/0 conn(0x7f2e840090e0 legacy=0x7f2e84009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:57.670 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 8979, 2024-09-17T15:51:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 3, 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:57.688 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:57.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2ea5e72640 1 -- 172.21.15.64:0/2324818425 >> v1:172.21.15.100:6812/4210963484 conn(0x7f2e80040d00 legacy=0x7f2e800431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2ea5e72640 10 monclient: shutdown 2024-09-17T15:51:57.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2ea5e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:57.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2ea5e72640 1 -- 172.21.15.64:0/2324818425 >> v1:172.21.15.64:6793/0 conn(0x7f2ea0135c80 legacy=0x7f2ea0131fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e9ffff640 1 -- 172.21.15.64:0/2324818425 reap_dead start 2024-09-17T15:51:57.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2e9ffff640 1 -- 172.21.15.64:0/2324818425 reap_dead start 2024-09-17T15:51:57.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2ea5e72640 1 -- 172.21.15.64:0/2324818425 shutdown_connections 2024-09-17T15:51:57.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.646+0000 7f2ea5e72640 1 -- 172.21.15.64:0/2324818425 >> 172.21.15.64:0/2324818425 conn(0x7f2ea006daa0 msgr2=0x7f2ea0133cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:57.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.647+0000 7f2ea5e72640 1 -- 172.21.15.64:0/2324818425 shutdown_connections 2024-09-17T15:51:57.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.647+0000 7f2ea5e72640 1 -- 172.21.15.64:0/2324818425 wait complete. 2024-09-17T15:51:57.691 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:57.691 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-17T15:51:57.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-17T15:51:57.752 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:57.752 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39740]: starting. 2024-09-17T15:51:57.752 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39740]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39584 2024-09-17T15:51:57.752 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39740]: shutting down. 2024-09-17T15:51:57.754 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:57.754 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:51:57.754 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:51:57.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:51:57.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.801+0000 7f8e2cb5d640 10 monclient: get_monmap_and_config 2024-09-17T15:51:57.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.801+0000 7f8e2cb5d640 10 monclient: build_initial_monmap 2024-09-17T15:51:57.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.801+0000 7f8e2cb5d640 10 monclient: monmap: 2024-09-17T15:51:57.803 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:57.803 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.803 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:57.802547+0000 2024-09-17T15:51:57.803 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:57.802547+0000 2024-09-17T15:51:57.804 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:57.804 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.804 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:57.804 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:57.804 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:57.804 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:57.804 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:57.804 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:57.804 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:57.804 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:57.804 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:57.804 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.802+0000 7f8e2cb5d640 1 Processor -- start 2024-09-17T15:51:57.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.802+0000 7f8e2cb5d640 1 -- start start 2024-09-17T15:51:57.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.802+0000 7f8e2cb5d640 10 monclient: init 2024-09-17T15:51:57.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.802+0000 7f8e2cb5d640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:57.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.802+0000 7f8e2cb5d640 10 monclient: _add_conns ranks=[7,0,2,1,4,3,6,8,5] 2024-09-17T15:51:57.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.802+0000 7f8e2cb5d640 10 monclient(hunting): picked mon.noname-i con 0x7f8e280a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:57.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.802+0000 7f8e2cb5d640 10 monclient(hunting): picked mon.noname-a con 0x7f8e28135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:57.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.802+0000 7f8e2cb5d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e28135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:57.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.802+0000 7f8e2cb5d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e2809d6f0 con 0x7f8e28135b50 2024-09-17T15:51:57.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.802+0000 7f8e2cb5d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e2809d9d0 con 0x7f8e28135f70 2024-09-17T15:51:57.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2cb5d640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e2809dcb0 con 0x7f8e280a5520 2024-09-17T15:51:57.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2cb5d640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:57.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2cb5d640 10 monclient(hunting): authenticate will time out at 1115.478018s 2024-09-17T15:51:57.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e25d9b640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f8e28135b50 0x7f8e2813deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:42630/0 (socket says 172.21.15.64:42630) 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e26d9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f8e28135f70 0x7f8e281415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51466/0 (socket says 172.21.15.64:51466) 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e25d9b640 1 -- 172.21.15.64:0/184816970 learned_addr learned my addr 172.21.15.64:0/184816970 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 -- 172.21.15.64:0/184816970 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2443944493 0 0) 0x7f8e2809d9d0 con 0x7f8e28135f70 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient(hunting): my global_id is 124913 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 -- 172.21.15.64:0/184816970 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e10003520 con 0x7f8e28135f70 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 -- 172.21.15.64:0/184816970 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1162658323 0 0) 0x7f8e2809d6f0 con 0x7f8e28135b50 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient(hunting): my global_id is 124830 2024-09-17T15:51:57.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 -- 172.21.15.64:0/184816970 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e2809d9d0 con 0x7f8e28135b50 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 -- 172.21.15.64:0/184816970 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1840260583 0 0) 0x7f8e2809dcb0 con 0x7f8e280a5520 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient(hunting): my global_id is 154249 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 -- 172.21.15.64:0/184816970 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e2809d6f0 con 0x7f8e280a5520 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 -- 172.21.15.64:0/184816970 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2111801434 0 0) 0x7f8e10003520 con 0x7f8e28135f70 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 -- 172.21.15.64:0/184816970 >> v1:172.21.15.100:6792/0 conn(0x7f8e280a5520 legacy=0x7f8e280a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 -- 172.21.15.64:0/184816970 >> v1:172.21.15.64:6789/0 conn(0x7f8e28135b50 legacy=0x7f8e2813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient: _finish_hunting 0 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 monclient: found mon.noname-b 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 -- 172.21.15.64:0/184816970 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e2809ccd0 con 0x7f8e28135f70 2024-09-17T15:51:57.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient: _finish_auth 0 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient: _check_auth_tickets 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 -- 172.21.15.64:0/184816970 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8e20001ae0 con 0x7f8e28135f70 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2cb5d640 5 monclient: authenticate success, global_id 124913 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient: dump: 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.816 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.817 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.817 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.817 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.817 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.817 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.817 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.817 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.817 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.817 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2cb5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:57.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 -- 172.21.15.64:0/184816970 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e20001d00 con 0x7f8e28135f70 2024-09-17T15:51:57.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:57.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 1 -- 172.21.15.64:0/184816970 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8e20004ea0 con 0x7f8e28135f70 2024-09-17T15:51:57.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2cb5d640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:57.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2cb5d640 4 set_mon_vals no callback set 2024-09-17T15:51:57.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:57.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.803+0000 7f8e2559a640 10 monclient: dump: 2024-09-17T15:51:57.818 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.818 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.818 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.818 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.818 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.818 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.818 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.818 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.819 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.819 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.819 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.819 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.819 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.819 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.819 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.819 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.804+0000 7f8e2cb5d640 10 monclient: shutdown 2024-09-17T15:51:57.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.804+0000 7f8e2cb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:57.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.804+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/184816970 >> v1:172.21.15.64:6790/0 conn(0x7f8e28135f70 legacy=0x7f8e281415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.804+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/184816970 shutdown_connections 2024-09-17T15:51:57.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.804+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/184816970 >> 172.21.15.64:0/184816970 conn(0x7f8e2806db30 msgr2=0x7f8e280ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:57.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.804+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/184816970 shutdown_connections 2024-09-17T15:51:57.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.804+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/184816970 wait complete. 2024-09-17T15:51:57.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.805+0000 7f8e2cb5d640 10 monclient: build_initial_monmap 2024-09-17T15:51:57.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.805+0000 7f8e2cb5d640 10 monclient: monmap: 2024-09-17T15:51:57.820 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:57.820 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:57.820 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:57.820 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:57.820 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:57.820 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:57.820 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:57.820 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.805+0000 7f8e2cb5d640 1 Processor -- start 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.805+0000 7f8e2cb5d640 1 -- start start 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.805+0000 7f8e2cb5d640 10 monclient: init 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.805+0000 7f8e2cb5d640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.805+0000 7f8e2cb5d640 10 monclient: _add_conns ranks=[6,1,0,4,5,2,8,7,3] 2024-09-17T15:51:57.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.805+0000 7f8e2cb5d640 10 monclient(hunting): picked mon.noname-d con 0x7f8e280a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:51:57.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.805+0000 7f8e2cb5d640 10 monclient(hunting): picked mon.noname-f con 0x7f8e28135b50 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:57.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.805+0000 7f8e2cb5d640 10 monclient(hunting): picked mon.noname-a con 0x7f8e28135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:57.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.805+0000 7f8e2cb5d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e280a5100 con 0x7f8e28135f70 2024-09-17T15:51:57.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e2cb5d640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e280a0210 con 0x7f8e28135b50 2024-09-17T15:51:57.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e2cb5d640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e280a04f0 con 0x7f8e280a5520 2024-09-17T15:51:57.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e2cb5d640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:57.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e2cb5d640 10 monclient(hunting): authenticate will time out at 1115.481040s 2024-09-17T15:51:57.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e26d9d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f8e28135f70 0x7f8e281d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:42646/0 (socket says 172.21.15.64:42646) 2024-09-17T15:51:57.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e26d9d640 1 -- 172.21.15.64:0/1257320069 learned_addr learned my addr 172.21.15.64:0/1257320069 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:57.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e2659c640 1 --1- 172.21.15.64:0/1257320069 >> v1:172.21.15.64:6792/0 conn(0x7f8e280a5520 0x7f8e28131920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:43834/0 (socket says 172.21.15.64:43834) 2024-09-17T15:51:57.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 840881683 0 0) 0x7f8e2809ccd0 con 0x7f8e28135f70 2024-09-17T15:51:57.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): my global_id is 124839 2024-09-17T15:51:57.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e08003520 con 0x7f8e28135f70 2024-09-17T15:51:57.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 335042326 0 0) 0x7f8e280a04f0 con 0x7f8e280a5520 2024-09-17T15:51:57.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): my global_id is 124800 2024-09-17T15:51:57.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e2809ccd0 con 0x7f8e280a5520 2024-09-17T15:51:57.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3566454463 0 0) 0x7f8e280a0210 con 0x7f8e28135b50 2024-09-17T15:51:57.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): my global_id is 6625 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e280a04f0 con 0x7f8e28135b50 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1996923166 0 0) 0x7f8e08003520 con 0x7f8e28135f70 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e280a0210 con 0x7f8e28135f70 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8e20001b60 con 0x7f8e28135f70 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): dump: 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.824 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 930215426 0 0) 0x7f8e2809ccd0 con 0x7f8e280a5520 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e08003520 con 0x7f8e280a5520 2024-09-17T15:51:57.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8e18001a10 con 0x7f8e280a5520 2024-09-17T15:51:57.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:57.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:51:57.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): dump: 2024-09-17T15:51:57.826 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.826 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.826 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.826 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.826 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.826 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.826 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.826 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.826 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.826 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.827 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.827 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.827 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.827 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.827 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.827 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3459576663 0 0) 0x7f8e280a04f0 con 0x7f8e28135b50 2024-09-17T15:51:57.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e2809ccd0 con 0x7f8e28135b50 2024-09-17T15:51:57.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8e1c001430 con 0x7f8e28135b50 2024-09-17T15:51:57.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:57.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:51:57.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient(hunting): dump: 2024-09-17T15:51:57.827 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.828 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.829 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3483028454 0 0) 0x7f8e280a0210 con 0x7f8e28135f70 2024-09-17T15:51:57.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 >> v1:172.21.15.64:6792/0 conn(0x7f8e280a5520 legacy=0x7f8e28131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 >> v1:172.21.15.100:6789/0 conn(0x7f8e28135b50 legacy=0x7f8e28132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:51:57.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 monclient: found mon.a 2024-09-17T15:51:57.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:57.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:51:57.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e28135310 con 0x7f8e28135f70 2024-09-17T15:51:57.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient: _finish_auth 0 2024-09-17T15:51:57.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:57.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:57.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e2cb5d640 5 monclient: authenticate success, global_id 124839 2024-09-17T15:51:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e2cb5d640 10 monclient: _renew_subs 2024-09-17T15:51:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e2cb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:51:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8e2809f960 con 0x7f8e28135f70 2024-09-17T15:51:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e2cb5d640 10 monclient: _renew_subs 2024-09-17T15:51:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e2cb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:51:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.806+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8e28132740 con 0x7f8e28135f70 2024-09-17T15:51:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.807+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e20001460 con 0x7f8e28135f70 2024-09-17T15:51:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.807+0000 7f8e16ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.807+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8e20005150 con 0x7f8e28135f70 2024-09-17T15:51:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.807+0000 7f8e16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.807+0000 7f8e24d99640 4 set_mon_vals no callback set 2024-09-17T15:51:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.807+0000 7f8e16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:57.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.807+0000 7f8e16ffd640 10 monclient: dump: 2024-09-17T15:51:57.831 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.831 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.831 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.831 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.831 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.831 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.831 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.831 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.831 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.831 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.831 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.831 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.831 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.807+0000 7f8e2cb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.807+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8e0c0065c0 con 0x7f8e0c002860 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.807+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f8e200052f0 con 0x7f8e28135f70 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(288..288 src has 192..288) ==== 3506+0+0 (unknown 2682836943 0 0) 0x7f8e2004ec30 con 0x7f8e28135f70 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 813817022 0 0) 0x7f8e0c0065c0 con 0x7f8e0c002860 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 10 monclient(hunting): my global_id is 124809 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e280a0210 con 0x7f8e0c002860 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e25d9b640 10 monclient: get_auth_request con 0x7f8e08040ca0 auth_method 0 2024-09-17T15:51:57.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3604540491 0 0) 0x7f8e280a0210 con 0x7f8e0c002860 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e0c0065c0 con 0x7f8e0c002860 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8e18003c00 con 0x7f8e0c002860 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 10 monclient: dump: 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.833 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.834 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.834 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.834 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.834 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.834 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.834 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3264738106 0 0) 0x7f8e0c0065c0 con 0x7f8e0c002860 2024-09-17T15:51:57.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e0c00a160 con 0x7f8e0c002860 2024-09-17T15:51:57.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8e18003e70 con 0x7f8e0c002860 2024-09-17T15:51:57.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:57.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:57.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 >> v1:172.21.15.64:6792/0 conn(0x7f8e0c002860 legacy=0x7f8e0c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.808+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 >> v1:172.21.15.64:6792/0 conn(0x7f8e0c002860 legacy=0x7f8e0c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:57.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e2cb5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:57.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8e0c009e00 con 0x7f8e0c0090e0 2024-09-17T15:51:57.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2046061109 0 0) 0x7f8e0c009e00 con 0x7f8e0c0090e0 2024-09-17T15:51:57.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 10 monclient(hunting): my global_id is 124818 2024-09-17T15:51:57.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e0c0065c0 con 0x7f8e0c0090e0 2024-09-17T15:51:57.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 351006197 0 0) 0x7f8e0c0065c0 con 0x7f8e0c0090e0 2024-09-17T15:51:57.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e0c009e00 con 0x7f8e0c0090e0 2024-09-17T15:51:57.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8e18005b00 con 0x7f8e0c0090e0 2024-09-17T15:51:57.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:51:57.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 10 monclient: dump: 2024-09-17T15:51:57.837 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.837 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.837 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.837 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.837 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.837 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.838 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.838 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.838 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.838 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.838 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.838 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.838 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.838 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.838 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.838 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1617530970 0 0) 0x7f8e0c009e00 con 0x7f8e0c0090e0 2024-09-17T15:51:57.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 --> v1:172.21.15.64:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8e0c00c3a0 con 0x7f8e0c0090e0 2024-09-17T15:51:57.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 2887994105) 0x7f8e18007d80 con 0x7f8e0c0090e0 2024-09-17T15:51:57.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:57.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:57.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 >> v1:172.21.15.64:6792/0 conn(0x7f8e0c0090e0 legacy=0x7f8e0c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.815+0000 7f8e16ffd640 1 -- 172.21.15.64:0/1257320069 >> v1:172.21.15.64:6792/0 conn(0x7f8e0c0090e0 legacy=0x7f8e0c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:57.839 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 124733, 2024-09-17T15:51:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:51:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:57.857 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:57.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.816+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/1257320069 >> v1:172.21.15.100:6812/4210963484 conn(0x7f8e08040ca0 legacy=0x7f8e08043160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.816+0000 7f8e2cb5d640 10 monclient: shutdown 2024-09-17T15:51:57.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.816+0000 7f8e2cb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:57.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.816+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/1257320069 >> v1:172.21.15.64:6789/0 conn(0x7f8e28135f70 legacy=0x7f8e281d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.816+0000 7f8e26d9d640 1 -- 172.21.15.64:0/1257320069 reap_dead start 2024-09-17T15:51:57.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.816+0000 7f8e26d9d640 1 -- 172.21.15.64:0/1257320069 reap_dead start 2024-09-17T15:51:57.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.816+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/1257320069 shutdown_connections 2024-09-17T15:51:57.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.816+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/1257320069 >> 172.21.15.64:0/1257320069 conn(0x7f8e2806db30 msgr2=0x7f8e2812c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:57.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.816+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/1257320069 shutdown_connections 2024-09-17T15:51:57.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.816+0000 7f8e2cb5d640 1 -- 172.21.15.64:0/1257320069 wait complete. 2024-09-17T15:51:57.859 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:57.860 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-17T15:51:57.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-17T15:51:57.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.961+0000 7f07eed44640 10 monclient: get_monmap_and_config 2024-09-17T15:51:57.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.961+0000 7f07eed44640 10 monclient: build_initial_monmap 2024-09-17T15:51:57.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.961+0000 7f07eed44640 10 monclient: monmap: 2024-09-17T15:51:57.963 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:57.963 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.963 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:57.962486+0000 2024-09-17T15:51:57.963 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:57.962486+0000 2024-09-17T15:51:57.964 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:57.964 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.964 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:57.964 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:57.964 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:57.964 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:57.964 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:57.964 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:57.964 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:57.964 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:57.964 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:57.964 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.962+0000 7f07eed44640 1 Processor -- start 2024-09-17T15:51:57.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.962+0000 7f07eed44640 1 -- start start 2024-09-17T15:51:57.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.962+0000 7f07eed44640 10 monclient: init 2024-09-17T15:51:57.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.962+0000 7f07eed44640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:57.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.962+0000 7f07eed44640 10 monclient: _add_conns ranks=[2,0,3,6,5,1,7,4,8] 2024-09-17T15:51:57.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.962+0000 7f07eed44640 10 monclient(hunting): picked mon.noname-b con 0x7f07e80a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:57.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.962+0000 7f07eed44640 10 monclient(hunting): picked mon.noname-a con 0x7f07e8135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:57.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.962+0000 7f07eed44640 10 monclient(hunting): picked mon.noname-g con 0x7f07e8135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:57.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.962+0000 7f07eed44640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e809d6f0 con 0x7f07e8135b50 2024-09-17T15:51:57.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.962+0000 7f07eed44640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e809d9d0 con 0x7f07e80a5520 2024-09-17T15:51:57.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.962+0000 7f07eed44640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e809dcb0 con 0x7f07e8135f70 2024-09-17T15:51:57.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.962+0000 7f07eed44640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:57.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.962+0000 7f07eed44640 10 monclient(hunting): authenticate will time out at 1115.637955s 2024-09-17T15:51:57.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07ecae0640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f07e80a5520 0x7f07e80a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51468/0 (socket says 172.21.15.64:51468) 2024-09-17T15:51:57.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07ecae0640 1 -- 172.21.15.64:0/2479929194 learned_addr learned my addr 172.21.15.64:0/2479929194 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:57.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e7fff640 1 --1- 172.21.15.64:0/2479929194 >> v1:172.21.15.64:6789/0 conn(0x7f07e8135b50 0x7f07e813deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:42658/0 (socket says 172.21.15.64:42658) 2024-09-17T15:51:57.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 -- 172.21.15.64:0/2479929194 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 546681826 0 0) 0x7f07e809d9d0 con 0x7f07e80a5520 2024-09-17T15:51:57.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient(hunting): my global_id is 124922 2024-09-17T15:51:57.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 -- 172.21.15.64:0/2479929194 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07d8003520 con 0x7f07e80a5520 2024-09-17T15:51:57.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 -- 172.21.15.64:0/2479929194 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2944490809 0 0) 0x7f07e809d6f0 con 0x7f07e8135b50 2024-09-17T15:51:57.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient(hunting): my global_id is 124857 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 -- 172.21.15.64:0/2479929194 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07e809d9d0 con 0x7f07e8135b50 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 -- 172.21.15.64:0/2479929194 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1360942124 0 0) 0x7f07e809dcb0 con 0x7f07e8135f70 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient(hunting): my global_id is 115563 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 -- 172.21.15.64:0/2479929194 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07e809d6f0 con 0x7f07e8135f70 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 -- 172.21.15.64:0/2479929194 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3879536308 0 0) 0x7f07d8003520 con 0x7f07e80a5520 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 -- 172.21.15.64:0/2479929194 >> v1:172.21.15.100:6790/0 conn(0x7f07e8135f70 legacy=0x7f07e81415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 -- 172.21.15.64:0/2479929194 >> v1:172.21.15.64:6789/0 conn(0x7f07e8135b50 legacy=0x7f07e813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:51:57.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 monclient: found mon.noname-b 2024-09-17T15:51:57.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:57.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:51:57.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 -- 172.21.15.64:0/2479929194 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e809ccd0 con 0x7f07e80a5520 2024-09-17T15:51:57.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:57.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:57.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:57.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 -- 172.21.15.64:0/2479929194 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f07dc001830 con 0x7f07e80a5520 2024-09-17T15:51:57.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07eed44640 5 monclient: authenticate success, global_id 124922 2024-09-17T15:51:57.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07eed44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:57.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:51:57.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient: dump: 2024-09-17T15:51:57.976 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.977 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.978 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07eed44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 -- 172.21.15.64:0/2479929194 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07dc001a50 con 0x7f07e80a5520 2024-09-17T15:51:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 1 -- 172.21.15.64:0/2479929194 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f07dc001d40 con 0x7f07e80a5520 2024-09-17T15:51:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07eed44640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07eed44640 4 set_mon_vals no callback set 2024-09-17T15:51:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.963+0000 7f07e77fe640 10 monclient: dump: 2024-09-17T15:51:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.979 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.979 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.979 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.979 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.979 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.979 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.979 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.979 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.979 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.979 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.979 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.979 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.980 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.964+0000 7f07eed44640 10 monclient: shutdown 2024-09-17T15:51:57.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.964+0000 7f07eed44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:57.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.964+0000 7f07eed44640 1 -- 172.21.15.64:0/2479929194 >> v1:172.21.15.64:6790/0 conn(0x7f07e80a5520 legacy=0x7f07e80a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.964+0000 7f07eed44640 1 -- 172.21.15.64:0/2479929194 shutdown_connections 2024-09-17T15:51:57.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.964+0000 7f07eed44640 1 -- 172.21.15.64:0/2479929194 >> 172.21.15.64:0/2479929194 conn(0x7f07e806db30 msgr2=0x7f07e80ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:57.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.964+0000 7f07eed44640 1 -- 172.21.15.64:0/2479929194 shutdown_connections 2024-09-17T15:51:57.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.964+0000 7f07eed44640 1 -- 172.21.15.64:0/2479929194 wait complete. 2024-09-17T15:51:57.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.964+0000 7f07eed44640 10 monclient: build_initial_monmap 2024-09-17T15:51:57.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.964+0000 7f07eed44640 10 monclient: monmap: 2024-09-17T15:51:57.980 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:57.980 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:57.980 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:57.981 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:57.981 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:57.981 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:57.981 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:57.981 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:57.981 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:57.981 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:57.981 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:57.981 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:57.981 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:57.981 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:57.981 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:57.981 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.965+0000 7f07eed44640 1 Processor -- start 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.965+0000 7f07eed44640 1 -- start start 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.965+0000 7f07eed44640 10 monclient: init 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.965+0000 7f07eed44640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.965+0000 7f07eed44640 10 monclient: _add_conns ranks=[2,8,3,7,6,5,0,4,1] 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.965+0000 7f07eed44640 10 monclient(hunting): picked mon.noname-b con 0x7f07e80a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.965+0000 7f07eed44640 10 monclient(hunting): picked mon.noname-e con 0x7f07e8135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.965+0000 7f07eed44640 10 monclient(hunting): picked mon.noname-g con 0x7f07e8135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07eed44640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e80a6710 con 0x7f07e80a5520 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07eed44640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e809e630 con 0x7f07e8135f70 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07eed44640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e809fa60 con 0x7f07e8135b50 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07ecae0640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f07e80a5520 0x7f07e8131bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51480/0 (socket says 172.21.15.64:51480) 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07eed44640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:57.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07ecae0640 1 -- 172.21.15.64:0/2929171577 learned_addr learned my addr 172.21.15.64:0/2929171577 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:57.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07eed44640 10 monclient(hunting): authenticate will time out at 1115.641104s 2024-09-17T15:51:57.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3550988908 0 0) 0x7f07e809fa60 con 0x7f07e8135b50 2024-09-17T15:51:57.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): my global_id is 154214 2024-09-17T15:51:57.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07cc003520 con 0x7f07e8135b50 2024-09-17T15:51:57.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 631703299 0 0) 0x7f07e809ccd0 con 0x7f07e80a5520 2024-09-17T15:51:57.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): my global_id is 124931 2024-09-17T15:51:57.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07e809fa60 con 0x7f07e80a5520 2024-09-17T15:51:57.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1849608624 0 0) 0x7f07e809e630 con 0x7f07e8135f70 2024-09-17T15:51:57.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): my global_id is 115572 2024-09-17T15:51:57.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07e809ccd0 con 0x7f07e8135f70 2024-09-17T15:51:57.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3378463517 0 0) 0x7f07cc003520 con 0x7f07e8135b50 2024-09-17T15:51:57.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07e809e630 con 0x7f07e8135b50 2024-09-17T15:51:57.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f07d4001570 con 0x7f07e8135b50 2024-09-17T15:51:57.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:57.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:51:57.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): dump: 2024-09-17T15:51:57.985 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.985 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.985 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.985 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.985 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.985 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.985 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.985 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.985 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.985 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.985 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.985 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3885157540 0 0) 0x7f07e809fa60 con 0x7f07e80a5520 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07cc003520 con 0x7f07e80a5520 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3547523855 0 0) 0x7f07e809ccd0 con 0x7f07e8135f70 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07e809fa60 con 0x7f07e8135f70 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f07dc0017f0 con 0x7f07e80a5520 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): dump: 2024-09-17T15:51:57.986 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.987 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.987 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.987 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.987 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.987 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.987 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.987 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.987 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.987 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.987 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.987 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.987 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.987 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.988 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.988 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f07e0001cf0 con 0x7f07e8135f70 2024-09-17T15:51:57.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:57.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:51:57.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient(hunting): dump: 2024-09-17T15:51:57.988 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.988 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.988 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.988 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.988 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.988 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.988 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2261320191 0 0) 0x7f07e809e630 con 0x7f07e8135b50 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 >> v1:172.21.15.100:6790/0 conn(0x7f07e8135f70 legacy=0x7f07e81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 >> v1:172.21.15.64:6790/0 conn(0x7f07e80a5520 legacy=0x7f07e8131bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient: _finish_hunting 0 2024-09-17T15:51:57.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 monclient: found mon.e 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e81329d0 con 0x7f07e8135b50 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient: _finish_auth 0 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07eed44640 5 monclient: authenticate success, global_id 154214 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07eed44640 10 monclient: _renew_subs 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07eed44640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07eed44640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f07e80a08d0 con 0x7f07e8135b50 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07eed44640 10 monclient: _renew_subs 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07eed44640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.966+0000 7f07eed44640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f07e80a5160 con 0x7f07e8135b50 2024-09-17T15:51:57.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.967+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07d4004a90 con 0x7f07e8135b50 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.967+0000 7f07e57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.967+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f07d4004f80 con 0x7f07e8135b50 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.967+0000 7f07e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.967+0000 7f07e57fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.967+0000 7f07e57fa640 10 monclient: dump: 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.991 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.967+0000 7f07e6ffd640 4 set_mon_vals no callback set 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.967+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f07d4011860 con 0x7f07e8135b50 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.967+0000 7f07eed44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.967+0000 7f07eed44640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f07c80065c0 con 0x7f07c8002860 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(288..288 src has 192..288) ==== 3506+0+0 (unknown 2682836943 0 0) 0x7f07d404eb50 con 0x7f07e8135b50 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07ed2e1640 10 monclient: get_auth_request con 0x7f07cc040d70 auth_method 0 2024-09-17T15:51:57.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3506989083 0 0) 0x7f07c80065c0 con 0x7f07c8002860 2024-09-17T15:51:57.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 10 monclient(hunting): my global_id is 154223 2024-09-17T15:51:57.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07e809e630 con 0x7f07c8002860 2024-09-17T15:51:57.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1029229890 0 0) 0x7f07e809e630 con 0x7f07c8002860 2024-09-17T15:51:57.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07c80065c0 con 0x7f07c8002860 2024-09-17T15:51:57.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f07dc004d40 con 0x7f07c8002860 2024-09-17T15:51:57.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:51:57.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 10 monclient: dump: 2024-09-17T15:51:57.993 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.993 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.994 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 54430303 0 0) 0x7f07c80065c0 con 0x7f07c8002860 2024-09-17T15:51:57.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07c800a160 con 0x7f07c8002860 2024-09-17T15:51:57.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f07dc007480 con 0x7f07c8002860 2024-09-17T15:51:57.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:57.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:57.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 >> v1:172.21.15.64:6793/0 conn(0x7f07c8002860 legacy=0x7f07c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.968+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 >> v1:172.21.15.64:6793/0 conn(0x7f07c8002860 legacy=0x7f07c8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:57.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07eed44640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:57.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07eed44640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f07c8009e20 con 0x7f07c80090e0 2024-09-17T15:51:57.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 614927831 0 0) 0x7f07c8009e20 con 0x7f07c80090e0 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 10 monclient(hunting): my global_id is 154232 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07c80065c0 con 0x7f07c80090e0 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4024900963 0 0) 0x7f07c80065c0 con 0x7f07c80090e0 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07c8009e20 con 0x7f07c80090e0 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f07dc005700 con 0x7f07c80090e0 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 10 monclient: dump: 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:57.997 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:57.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3722234614 0 0) 0x7f07c8009e20 con 0x7f07c80090e0 2024-09-17T15:51:57.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.975+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 --> v1:172.21.15.64:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f07c800c3f0 con 0x7f07c80090e0 2024-09-17T15:51:57.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 29677866) 0x7f07dc006bd0 con 0x7f07c80090e0 2024-09-17T15:51:57.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07e57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:57.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:57.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 >> v1:172.21.15.64:6793/0 conn(0x7f07c80090e0 legacy=0x7f07c8009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:57.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07e57fa640 1 -- 172.21.15.64:0/2929171577 >> v1:172.21.15.64:6793/0 conn(0x7f07c80090e0 legacy=0x7f07c8009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:57.999 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:57.999 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 9311, 2024-09-17T15:51:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:58.000 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.001 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.002 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.004 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:58.005 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.007 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:58.011 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.013 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:58.014 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.016 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:58.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:51:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.018 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:58.018 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07eed44640 1 -- 172.21.15.64:0/2929171577 >> v1:172.21.15.100:6812/4210963484 conn(0x7f07cc040d70 legacy=0x7f07cc043230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07eed44640 10 monclient: shutdown 2024-09-17T15:51:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07eed44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07eed44640 1 -- 172.21.15.64:0/2929171577 >> v1:172.21.15.64:6793/0 conn(0x7f07e8135b50 legacy=0x7f07e81322c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07ed2e1640 1 -- 172.21.15.64:0/2929171577 reap_dead start 2024-09-17T15:51:58.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07ed2e1640 1 -- 172.21.15.64:0/2929171577 reap_dead start 2024-09-17T15:51:58.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07eed44640 1 -- 172.21.15.64:0/2929171577 shutdown_connections 2024-09-17T15:51:58.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07eed44640 1 -- 172.21.15.64:0/2929171577 >> 172.21.15.64:0/2929171577 conn(0x7f07e806db30 msgr2=0x7f07e80a75d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:58.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07eed44640 1 -- 172.21.15.64:0/2929171577 shutdown_connections 2024-09-17T15:51:58.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:57.976+0000 7f07eed44640 1 -- 172.21.15.64:0/2929171577 wait complete. 2024-09-17T15:51:58.020 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:58.020 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-17T15:51:58.020 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-17T15:51:58.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.121+0000 7f5af5189640 10 monclient: get_monmap_and_config 2024-09-17T15:51:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.121+0000 7f5af5189640 10 monclient: build_initial_monmap 2024-09-17T15:51:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.121+0000 7f5af5189640 10 monclient: monmap: 2024-09-17T15:51:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:58.121889+0000 2024-09-17T15:51:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:58.121889+0000 2024-09-17T15:51:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:58.123 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:58.124 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:58.124 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.121+0000 7f5af5189640 1 Processor -- start 2024-09-17T15:51:58.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.121+0000 7f5af5189640 1 -- start start 2024-09-17T15:51:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.121+0000 7f5af5189640 10 monclient: init 2024-09-17T15:51:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 10 monclient: _add_conns ranks=[2,7,1,0,3,8,5,4,6] 2024-09-17T15:51:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 10 monclient(hunting): picked mon.noname-b con 0x7f5af00a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 10 monclient(hunting): picked mon.noname-i con 0x7f5af0135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 10 monclient(hunting): picked mon.noname-f con 0x7f5af0135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af009d6f0 con 0x7f5af0135f70 2024-09-17T15:51:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af009d9d0 con 0x7f5af00a5520 2024-09-17T15:51:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af009dcb0 con 0x7f5af0135b50 2024-09-17T15:51:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 10 monclient(hunting): authenticate will time out at 1115.797261s 2024-09-17T15:51:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aeed9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f5af00a5520 0x7f5af00a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51494/0 (socket says 172.21.15.64:51494) 2024-09-17T15:51:58.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aeed9d640 1 -- 172.21.15.64:0/3956716699 learned_addr learned my addr 172.21.15.64:0/3956716699 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 -- 172.21.15.64:0/3956716699 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2161529436 0 0) 0x7f5af009d9d0 con 0x7f5af00a5520 2024-09-17T15:51:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient(hunting): my global_id is 124940 2024-09-17T15:51:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 -- 172.21.15.64:0/3956716699 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5adc003520 con 0x7f5af00a5520 2024-09-17T15:51:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 -- 172.21.15.64:0/3956716699 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2648664769 0 0) 0x7f5af009dcb0 con 0x7f5af0135b50 2024-09-17T15:51:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient(hunting): my global_id is 154285 2024-09-17T15:51:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 -- 172.21.15.64:0/3956716699 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5af009d9d0 con 0x7f5af0135b50 2024-09-17T15:51:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 -- 172.21.15.64:0/3956716699 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2449572483 0 0) 0x7f5af009d6f0 con 0x7f5af0135f70 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient(hunting): my global_id is 6643 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 -- 172.21.15.64:0/3956716699 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5af009dcb0 con 0x7f5af0135f70 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 -- 172.21.15.64:0/3956716699 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 193066788 0 0) 0x7f5adc003520 con 0x7f5af00a5520 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 -- 172.21.15.64:0/3956716699 >> v1:172.21.15.100:6792/0 conn(0x7f5af0135b50 legacy=0x7f5af013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 -- 172.21.15.64:0/3956716699 >> v1:172.21.15.100:6789/0 conn(0x7f5af0135f70 legacy=0x7f5af01415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 monclient: found mon.noname-b 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 -- 172.21.15.64:0/3956716699 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af009ccd0 con 0x7f5af00a5520 2024-09-17T15:51:58.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient: _finish_auth 0 2024-09-17T15:51:58.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient: _check_auth_tickets 2024-09-17T15:51:58.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:58.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 5 monclient: authenticate success, global_id 124940 2024-09-17T15:51:58.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 -- 172.21.15.64:0/3956716699 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5ad8001830 con 0x7f5af00a5520 2024-09-17T15:51:58.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:51:58.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient: dump: 2024-09-17T15:51:58.136 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.136 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.136 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.136 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.136 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.136 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.137 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.137 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.137 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.137 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.137 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.137 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.137 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.137 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.137 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.137 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:58.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 -- 172.21.15.64:0/3956716699 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ad8001a50 con 0x7f5af00a5520 2024-09-17T15:51:58.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 1 -- 172.21.15.64:0/3956716699 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5ad8004be0 con 0x7f5af00a5520 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5af5189640 4 set_mon_vals no callback set 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.122+0000 7f5aedd9b640 10 monclient: dump: 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.138 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.123+0000 7f5af5189640 10 monclient: shutdown 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.123+0000 7f5af5189640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.123+0000 7f5af5189640 1 -- 172.21.15.64:0/3956716699 >> v1:172.21.15.64:6790/0 conn(0x7f5af00a5520 legacy=0x7f5af00a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.123+0000 7f5af5189640 1 -- 172.21.15.64:0/3956716699 shutdown_connections 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.123+0000 7f5af5189640 1 -- 172.21.15.64:0/3956716699 >> 172.21.15.64:0/3956716699 conn(0x7f5af006db30 msgr2=0x7f5af00ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:58.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.123+0000 7f5af5189640 1 -- 172.21.15.64:0/3956716699 shutdown_connections 2024-09-17T15:51:58.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.123+0000 7f5af5189640 1 -- 172.21.15.64:0/3956716699 wait complete. 2024-09-17T15:51:58.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.124+0000 7f5af5189640 10 monclient: build_initial_monmap 2024-09-17T15:51:58.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.124+0000 7f5af5189640 10 monclient: monmap: 2024-09-17T15:51:58.140 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:58.140 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:58.140 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:58.140 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:58.140 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:58.140 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:58.140 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:58.140 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:58.140 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:58.140 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:58.141 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:58.141 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:58.141 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:58.141 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:58.141 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:58.141 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.124+0000 7f5af5189640 1 Processor -- start 2024-09-17T15:51:58.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.124+0000 7f5af5189640 1 -- start start 2024-09-17T15:51:58.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.124+0000 7f5af5189640 10 monclient: init 2024-09-17T15:51:58.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5af5189640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:58.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5af5189640 10 monclient: _add_conns ranks=[5,3,4,6,8,2,1,0,7] 2024-09-17T15:51:58.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5af5189640 10 monclient(hunting): picked mon.noname-h con 0x7f5af00a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:58.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5af5189640 10 monclient(hunting): picked mon.noname-g con 0x7f5af0135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5af5189640 10 monclient(hunting): picked mon.noname-c con 0x7f5af0135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5af5189640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af0141a20 con 0x7f5af0135b50 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5af5189640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af013ad20 con 0x7f5af0135f70 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5af5189640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af00a63e0 con 0x7f5af00a5520 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5af5189640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5af5189640 10 monclient(hunting): authenticate will time out at 1115.800220s 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5aef59e640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f5af0135f70 0x7f5af01d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:57894/0 (socket says 172.21.15.64:57894) 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5aef59e640 1 -- 172.21.15.64:0/3492474319 learned_addr learned my addr 172.21.15.64:0/3492474319 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1895725724 0 0) 0x7f5af013ad20 con 0x7f5af0135f70 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): my global_id is 154264 2024-09-17T15:51:58.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ad0003520 con 0x7f5af0135f70 2024-09-17T15:51:58.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1947422426 0 0) 0x7f5af0141a20 con 0x7f5af0135b50 2024-09-17T15:51:58.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): my global_id is 115581 2024-09-17T15:51:58.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5af013ad20 con 0x7f5af0135b50 2024-09-17T15:51:58.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3608353772 0 0) 0x7f5af00a63e0 con 0x7f5af00a5520 2024-09-17T15:51:58.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): my global_id is 154274 2024-09-17T15:51:58.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5af0141a20 con 0x7f5af00a5520 2024-09-17T15:51:58.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4199202918 0 0) 0x7f5ad0003520 con 0x7f5af0135f70 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5af00a63e0 con 0x7f5af0135f70 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5ae8001db0 con 0x7f5af0135f70 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): dump: 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.144 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.145 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.145 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.145 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.145 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.145 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.145 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.145 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:58.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4273140076 0 0) 0x7f5af013ad20 con 0x7f5af0135b50 2024-09-17T15:51:58.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ad0003520 con 0x7f5af0135b50 2024-09-17T15:51:58.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5ae0001ad0 con 0x7f5af0135b50 2024-09-17T15:51:58.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:58.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): dump: 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.146 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.147 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.147 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.147 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1600786159 0 0) 0x7f5af0141a20 con 0x7f5af00a5520 2024-09-17T15:51:58.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5af013ad20 con 0x7f5af00a5520 2024-09-17T15:51:58.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5ad80010d0 con 0x7f5af00a5520 2024-09-17T15:51:58.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:58.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:51:58.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.125+0000 7f5ae77fe640 10 monclient(hunting): dump: 2024-09-17T15:51:58.147 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.147 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.147 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.148 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.148 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.148 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.148 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.148 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.148 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.148 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.148 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.148 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.148 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.148 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.148 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.148 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3733430919 0 0) 0x7f5af00a63e0 con 0x7f5af0135f70 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 >> v1:172.21.15.100:6791/0 conn(0x7f5af00a5520 legacy=0x7f5af0136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 >> v1:172.21.15.100:6790/0 conn(0x7f5af0135b50 legacy=0x7f5af01376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 1 monclient: found mon.c 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af0135070 con 0x7f5af0135f70 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5af5189640 5 monclient: authenticate success, global_id 154264 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5af5189640 10 monclient: _renew_subs 2024-09-17T15:51:58.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5af5189640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5af5189640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5af01389c0 con 0x7f5af0135f70 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5af5189640 10 monclient: _renew_subs 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5af5189640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5af5189640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f5af013a480 con 0x7f5af0135f70 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ae80020b0 con 0x7f5af0135f70 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5ae8003f80 con 0x7f5af0135f70 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5ae77fe640 10 monclient: dump: 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.150 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5aed59a640 4 set_mon_vals no callback set 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f5ae8004120 con 0x7f5af0135f70 2024-09-17T15:51:58.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.126+0000 7f5af5189640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5af5189640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5ad40065c0 con 0x7f5ad4002860 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(288..288 src has 192..288) ==== 3506+0+0 (unknown 2682836943 0 0) 0x7f5ae804ecb0 con 0x7f5af0135f70 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5aee59c640 10 monclient: get_auth_request con 0x7f5ad0040d50 auth_method 0 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3130137201 0 0) 0x7f5ad40065c0 con 0x7f5ad4002860 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 10 monclient(hunting): my global_id is 6652 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5af00a63e0 con 0x7f5ad4002860 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3694440919 0 0) 0x7f5af00a63e0 con 0x7f5ad4002860 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ad40065c0 con 0x7f5ad4002860 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5ad8004ea0 con 0x7f5ad4002860 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:51:58.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 10 monclient: dump: 2024-09-17T15:51:58.153 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.153 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.153 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.153 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.153 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.153 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.153 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.153 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.153 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.153 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.153 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.153 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.154 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.154 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.154 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.154 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3988173762 0 0) 0x7f5ad40065c0 con 0x7f5ad4002860 2024-09-17T15:51:58.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.127+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.100:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ad400a160 con 0x7f5ad4002860 2024-09-17T15:51:58.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.128+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5ad8007480 con 0x7f5ad4002860 2024-09-17T15:51:58.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.128+0000 7f5ae77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:58.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.128+0000 7f5ae77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:58.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.128+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 >> v1:172.21.15.100:6789/0 conn(0x7f5ad4002860 legacy=0x7f5ad4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.128+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 >> v1:172.21.15.100:6789/0 conn(0x7f5ad4002860 legacy=0x7f5ad4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:58.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5af5189640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:58.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5af5189640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5ad4009e70 con 0x7f5ad4009130 2024-09-17T15:51:58.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 333279008 0 0) 0x7f5ad4009e70 con 0x7f5ad4009130 2024-09-17T15:51:58.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5ae77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5ae77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5ae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5ae77fe640 10 monclient(hunting): my global_id is 6661 2024-09-17T15:51:58.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ad40065c0 con 0x7f5ad4009130 2024-09-17T15:51:58.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1189273211 0 0) 0x7f5ad40065c0 con 0x7f5ad4009130 2024-09-17T15:51:58.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ad4009e70 con 0x7f5ad4009130 2024-09-17T15:51:58.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5ad80056c0 con 0x7f5ad4009130 2024-09-17T15:51:58.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5ae77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5ae77fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:51:58.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.134+0000 7f5ae77fe640 10 monclient: dump: 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.157 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.158 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.158 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1077711845 0 0) 0x7f5ad4009e70 con 0x7f5ad4009130 2024-09-17T15:51:58.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 --> v1:172.21.15.100:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5ad400c410 con 0x7f5ad4009130 2024-09-17T15:51:58.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7880 (unknown 0 0 1630233885) 0x7f5ad8006b70 con 0x7f5ad4009130 2024-09-17T15:51:58.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5ae77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:58.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5ae77fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:58.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 >> v1:172.21.15.100:6789/0 conn(0x7f5ad4009130 legacy=0x7f5ad40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5ae77fe640 1 -- 172.21.15.64:0/3492474319 >> v1:172.21.15.100:6789/0 conn(0x7f5ad4009130 legacy=0x7f5ad40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:58.158 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:58.158 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:58.159 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:58.159 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "leader", 2024-09-17T15:51:58.159 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:58.159 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 374785, 2024-09-17T15:51:58.159 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:58.159 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:58.159 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:58.159 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:58.159 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:58.159 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:58.159 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:58.159 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:58.159 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.160 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.161 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:58.162 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.163 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.165 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.165 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:58.165 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.165 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.165 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.165 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:58.165 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:58.165 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.165 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.165 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:58.166 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.167 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:58.169 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.171 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:58.171 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:58.171 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.171 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.171 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:58.171 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.171 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.171 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:58.172 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.173 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:58.175 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.176 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.177 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.178 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": [ 2024-09-17T15:51:58.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.178 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:58.178 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:58.178 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:58.178 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.178 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.178 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:58.178 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:58.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5af5189640 1 -- 172.21.15.64:0/3492474319 >> v1:172.21.15.100:6812/4210963484 conn(0x7f5ad0040d50 legacy=0x7f5ad0043210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5af5189640 10 monclient: shutdown 2024-09-17T15:51:58.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5af5189640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:58.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5af5189640 1 -- 172.21.15.64:0/3492474319 >> v1:172.21.15.64:6791/0 conn(0x7f5af0135f70 legacy=0x7f5af01d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5aef59e640 1 -- 172.21.15.64:0/3492474319 reap_dead start 2024-09-17T15:51:58.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5aef59e640 1 -- 172.21.15.64:0/3492474319 reap_dead start 2024-09-17T15:51:58.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5af5189640 1 -- 172.21.15.64:0/3492474319 shutdown_connections 2024-09-17T15:51:58.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5af5189640 1 -- 172.21.15.64:0/3492474319 >> 172.21.15.64:0/3492474319 conn(0x7f5af006db30 msgr2=0x7f5af0133fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:58.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5af5189640 1 -- 172.21.15.64:0/3492474319 shutdown_connections 2024-09-17T15:51:58.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.135+0000 7f5af5189640 1 -- 172.21.15.64:0/3492474319 wait complete. 2024-09-17T15:51:58.180 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:58.180 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-17T15:51:58.181 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-17T15:51:58.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.281+0000 7fd036d53640 10 monclient: get_monmap_and_config 2024-09-17T15:51:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.281+0000 7fd036d53640 10 monclient: build_initial_monmap 2024-09-17T15:51:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.281+0000 7fd036d53640 10 monclient: monmap: 2024-09-17T15:51:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:58.282065+0000 2024-09-17T15:51:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:58.282065+0000 2024-09-17T15:51:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:58.284 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:58.284 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:58.284 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:58.284 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:58.284 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:58.284 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.281+0000 7fd036d53640 1 Processor -- start 2024-09-17T15:51:58.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd036d53640 1 -- start start 2024-09-17T15:51:58.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd036d53640 10 monclient: init 2024-09-17T15:51:58.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd036d53640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:58.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd036d53640 10 monclient: _add_conns ranks=[5,0,1,6,7,3,4,8,2] 2024-09-17T15:51:58.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd036d53640 10 monclient(hunting): picked mon.noname-h con 0x7fd0300a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:58.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd036d53640 10 monclient(hunting): picked mon.noname-a con 0x7fd030135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:58.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd036d53640 10 monclient(hunting): picked mon.noname-f con 0x7fd030135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:58.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd036d53640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03009d6f0 con 0x7fd030135b50 2024-09-17T15:51:58.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd036d53640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03009d9d0 con 0x7fd030135f70 2024-09-17T15:51:58.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd036d53640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03009dcb0 con 0x7fd0300a5520 2024-09-17T15:51:58.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd036d53640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd036d53640 10 monclient(hunting): authenticate will time out at 1115.957555s 2024-09-17T15:51:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd02ffff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fd030135b50 0x7fd03013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:42670/0 (socket says 172.21.15.64:42670) 2024-09-17T15:51:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd02ffff640 1 -- 172.21.15.64:0/3628516429 learned_addr learned my addr 172.21.15.64:0/3628516429 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd02f7fe640 1 -- 172.21.15.64:0/3628516429 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3398688520 0 0) 0x7fd03009d6f0 con 0x7fd030135b50 2024-09-17T15:51:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd02f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd02f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd02f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd02f7fe640 10 monclient(hunting): my global_id is 124866 2024-09-17T15:51:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.282+0000 7fd02f7fe640 1 -- 172.21.15.64:0/3628516429 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd020003520 con 0x7fd030135b50 2024-09-17T15:51:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 1 -- 172.21.15.64:0/3628516429 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1510464139 0 0) 0x7fd03009dcb0 con 0x7fd0300a5520 2024-09-17T15:51:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient(hunting): my global_id is 154283 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 1 -- 172.21.15.64:0/3628516429 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd03009d6f0 con 0x7fd0300a5520 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 1 -- 172.21.15.64:0/3628516429 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 215482919 0 0) 0x7fd03009d9d0 con 0x7fd030135f70 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient(hunting): my global_id is 6670 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 1 -- 172.21.15.64:0/3628516429 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd03009dcb0 con 0x7fd030135f70 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 1 -- 172.21.15.64:0/3628516429 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 89177262 0 0) 0x7fd020003520 con 0x7fd030135b50 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 1 -- 172.21.15.64:0/3628516429 >> v1:172.21.15.100:6791/0 conn(0x7fd0300a5520 legacy=0x7fd0300a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 1 -- 172.21.15.64:0/3628516429 >> v1:172.21.15.100:6789/0 conn(0x7fd030135f70 legacy=0x7fd0301415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 1 monclient: found mon.noname-a 2024-09-17T15:51:58.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 1 -- 172.21.15.64:0/3628516429 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03009ccd0 con 0x7fd030135b50 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd036d53640 5 monclient: authenticate success, global_id 124866 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 1 -- 172.21.15.64:0/3628516429 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd01c001830 con 0x7fd030135b50 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd036d53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient: dump: 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd036d53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 1 -- 172.21.15.64:0/3628516429 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd01c001a50 con 0x7fd030135b50 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd036d53640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 1 -- 172.21.15.64:0/3628516429 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd01c004be0 con 0x7fd030135b50 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd036d53640 4 set_mon_vals no callback set 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd02f7fe640 10 monclient: dump: 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.297 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.298 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.298 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.298 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.298 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.298 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.298 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.298 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.298 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.298 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.298 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd036d53640 10 monclient: shutdown 2024-09-17T15:51:58.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd036d53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:58.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd036d53640 1 -- 172.21.15.64:0/3628516429 >> v1:172.21.15.64:6789/0 conn(0x7fd030135b50 legacy=0x7fd03013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd036d53640 1 -- 172.21.15.64:0/3628516429 shutdown_connections 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd036d53640 1 -- 172.21.15.64:0/3628516429 >> 172.21.15.64:0/3628516429 conn(0x7fd03006db30 msgr2=0x7fd0300ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd036d53640 1 -- 172.21.15.64:0/3628516429 shutdown_connections 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.283+0000 7fd036d53640 1 -- 172.21.15.64:0/3628516429 wait complete. 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.284+0000 7fd036d53640 10 monclient: build_initial_monmap 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.284+0000 7fd036d53640 10 monclient: monmap: 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:58.299 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:58.300 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:58.300 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:58.300 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:58.300 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:58.300 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:58.300 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:58.300 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:58.300 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.284+0000 7fd036d53640 1 Processor -- start 2024-09-17T15:51:58.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.284+0000 7fd036d53640 1 -- start start 2024-09-17T15:51:58.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.284+0000 7fd036d53640 10 monclient: init 2024-09-17T15:51:58.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd036d53640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:58.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd036d53640 10 monclient: _add_conns ranks=[1,6,2,0,4,7,8,3,5] 2024-09-17T15:51:58.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd036d53640 10 monclient(hunting): picked mon.noname-f con 0x7fd0300a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:58.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd036d53640 10 monclient(hunting): picked mon.noname-d con 0x7fd030135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:51:58.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd036d53640 10 monclient(hunting): picked mon.noname-b con 0x7fd030135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:58.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd036d53640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd030134840 con 0x7fd0300a5520 2024-09-17T15:51:58.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd036d53640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd030138a90 con 0x7fd030135f70 2024-09-17T15:51:58.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd036d53640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd030137e10 con 0x7fd030135b50 2024-09-17T15:51:58.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd036d53640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:58.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd036d53640 10 monclient(hunting): authenticate will time out at 1115.960418s 2024-09-17T15:51:58.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02ffff640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fd030135b50 0x7fd030137700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:43854/0 (socket says 172.21.15.64:43854) 2024-09-17T15:51:58.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02ffff640 1 -- 172.21.15.64:0/2866528300 learned_addr learned my addr 172.21.15.64:0/2866528300 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:58.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd0352f0640 1 --1- 172.21.15.64:0/2866528300 >> v1:172.21.15.64:6790/0 conn(0x7fd030135f70 0x7fd0301d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51500/0 (socket says 172.21.15.64:51500) 2024-09-17T15:51:58.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4069733303 0 0) 0x7fd030137e10 con 0x7fd030135b50 2024-09-17T15:51:58.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 10 monclient(hunting): my global_id is 124845 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd014003520 con 0x7fd030135b50 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 879312280 0 0) 0x7fd030138a90 con 0x7fd030135f70 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 10 monclient(hunting): my global_id is 124949 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd030137e10 con 0x7fd030135f70 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 574230612 0 0) 0x7fd030134840 con 0x7fd0300a5520 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 10 monclient(hunting): my global_id is 6679 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.285+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd030138a90 con 0x7fd0300a5520 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2469536650 0 0) 0x7fd014003520 con 0x7fd030135b50 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd030134840 con 0x7fd030135b50 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3146172637 0 0) 0x7fd030137e10 con 0x7fd030135f70 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd014003520 con 0x7fd030135f70 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd01c004b20 con 0x7fd030135b50 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient(hunting): dump: 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.303 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:58.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd028001cf0 con 0x7fd030135f70 2024-09-17T15:51:58.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:58.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:58.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient(hunting): dump: 2024-09-17T15:51:58.305 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.305 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.305 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.305 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.305 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.305 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.305 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.305 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.305 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.305 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 165145657 0 0) 0x7fd030138a90 con 0x7fd0300a5520 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd030137e10 con 0x7fd0300a5520 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd024001a50 con 0x7fd0300a5520 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient(hunting): dump: 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.306 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.307 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3476268513 0 0) 0x7fd014003520 con 0x7fd030135f70 2024-09-17T15:51:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 >> v1:172.21.15.64:6792/0 conn(0x7fd030135b50 legacy=0x7fd030137700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 >> v1:172.21.15.100:6789/0 conn(0x7fd0300a5520 legacy=0x7fd030136ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient: _finish_hunting 0 2024-09-17T15:51:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 monclient: found mon.b 2024-09-17T15:51:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:51:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03013a550 con 0x7fd030135f70 2024-09-17T15:51:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:51:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:51:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd036d53640 5 monclient: authenticate success, global_id 124949 2024-09-17T15:51:58.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd036d53640 10 monclient: _renew_subs 2024-09-17T15:51:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd036d53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:51:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd036d53640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd03013a830 con 0x7fd030135f70 2024-09-17T15:51:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd036d53640 10 monclient: _renew_subs 2024-09-17T15:51:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd036d53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:51:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd036d53640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd030142440 con 0x7fd030135f70 2024-09-17T15:51:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd028002030 con 0x7fd030135f70 2024-09-17T15:51:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd028003f80 con 0x7fd030135f70 2024-09-17T15:51:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02effd640 4 set_mon_vals no callback set 2024-09-17T15:51:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.286+0000 7fd02d7fa640 10 monclient: dump: 2024-09-17T15:51:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd036d53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd036d53640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd0100065c0 con 0x7fd010002860 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fd0280041a0 con 0x7fd030135f70 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(288..288 src has 192..288) ==== 3506+0+0 (unknown 2682836943 0 0) 0x7fd02804ead0 con 0x7fd030135f70 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02ffff640 10 monclient: get_auth_request con 0x7fd014040c70 auth_method 0 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 725914171 0 0) 0x7fd0100065c0 con 0x7fd010002860 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 10 monclient(hunting): my global_id is 115590 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd014003520 con 0x7fd010002860 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1961512441 0 0) 0x7fd014003520 con 0x7fd010002860 2024-09-17T15:51:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd0100065c0 con 0x7fd010002860 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd024003950 con 0x7fd010002860 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.287+0000 7fd02d7fa640 10 monclient: dump: 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.313 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.288+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 710950413 0 0) 0x7fd0100065c0 con 0x7fd010002860 2024-09-17T15:51:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.288+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.100:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd01000a160 con 0x7fd010002860 2024-09-17T15:51:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.288+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd024005b30 con 0x7fd010002860 2024-09-17T15:51:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.288+0000 7fd02d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.288+0000 7fd02d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.288+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 >> v1:172.21.15.100:6790/0 conn(0x7fd010002860 legacy=0x7fd010004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.288+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 >> v1:172.21.15.100:6790/0 conn(0x7fd010002860 legacy=0x7fd010004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.294+0000 7fd036d53640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.294+0000 7fd036d53640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd010009e20 con 0x7fd0100090e0 2024-09-17T15:51:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 713908911 0 0) 0x7fd010009e20 con 0x7fd0100090e0 2024-09-17T15:51:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 10 monclient(hunting): my global_id is 115599 2024-09-17T15:51:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0100065c0 con 0x7fd0100090e0 2024-09-17T15:51:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3116008504 0 0) 0x7fd0100065c0 con 0x7fd0100090e0 2024-09-17T15:51:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd010009e20 con 0x7fd0100090e0 2024-09-17T15:51:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd0240042a0 con 0x7fd0100090e0 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 10 monclient: dump: 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.317 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.317 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.317 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.317 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.317 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4116607787 0 0) 0x7fd010009e20 con 0x7fd0100090e0 2024-09-17T15:51:58.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 --> v1:172.21.15.100:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd01000c3f0 con 0x7fd0100090e0 2024-09-17T15:51:58.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1091771481) 0x7fd024004500 con 0x7fd0100090e0 2024-09-17T15:51:58.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:58.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:58.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 >> v1:172.21.15.100:6790/0 conn(0x7fd0100090e0 legacy=0x7fd010009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.295+0000 7fd02d7fa640 1 -- 172.21.15.64:0/2866528300 >> v1:172.21.15.100:6790/0 conn(0x7fd0100090e0 legacy=0x7fd010009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:58.318 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:58.318 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:58.318 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:58.318 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:58.318 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:58.318 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 241518, 2024-09-17T15:51:58.318 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:58.318 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:58.318 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:58.318 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:58.318 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:58.318 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:58.319 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:58.319 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:58.319 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:58.319 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:58.319 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.319 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:51:58.319 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:58.319 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:58.319 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:58.319 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.319 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.319 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.319 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.320 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:58.321 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:58.322 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:58.322 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:58.322 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:58.322 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:58.322 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:58.322 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:58.322 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:58.322 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:58.322 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:58.322 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:58.322 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:58.322 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.322 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.323 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:58.324 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:58.324 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.324 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.324 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.324 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:58.324 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.324 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.324 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.324 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:58.324 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:58.324 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.325 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.326 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:58.327 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.328 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.329 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:58.330 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.331 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.334 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:58.335 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:58.336 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:58.337 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:51:58.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.337 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.337 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:58.337 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:58.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.296+0000 7fd036d53640 1 -- 172.21.15.64:0/2866528300 >> v1:172.21.15.100:6812/4210963484 conn(0x7fd014040c70 legacy=0x7fd014043130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.296+0000 7fd036d53640 10 monclient: shutdown 2024-09-17T15:51:58.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.296+0000 7fd036d53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:58.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.296+0000 7fd036d53640 1 -- 172.21.15.64:0/2866528300 >> v1:172.21.15.64:6790/0 conn(0x7fd030135f70 legacy=0x7fd0301d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.296+0000 7fd0352f0640 1 -- 172.21.15.64:0/2866528300 reap_dead start 2024-09-17T15:51:58.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.296+0000 7fd0352f0640 1 -- 172.21.15.64:0/2866528300 reap_dead start 2024-09-17T15:51:58.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.296+0000 7fd036d53640 1 -- 172.21.15.64:0/2866528300 shutdown_connections 2024-09-17T15:51:58.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.296+0000 7fd036d53640 1 -- 172.21.15.64:0/2866528300 >> 172.21.15.64:0/2866528300 conn(0x7fd03006db30 msgr2=0x7fd0300a4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:58.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.296+0000 7fd036d53640 1 -- 172.21.15.64:0/2866528300 shutdown_connections 2024-09-17T15:51:58.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.296+0000 7fd036d53640 1 -- 172.21.15.64:0/2866528300 wait complete. 2024-09-17T15:51:58.339 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:58.339 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-17T15:51:58.339 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-17T15:51:58.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.443+0000 7ffb5be7c640 10 monclient: get_monmap_and_config 2024-09-17T15:51:58.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.443+0000 7ffb5be7c640 10 monclient: build_initial_monmap 2024-09-17T15:51:58.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.443+0000 7ffb5be7c640 10 monclient: monmap: 2024-09-17T15:51:58.446 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:58.446 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.446 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:58.444302+0000 2024-09-17T15:51:58.446 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:58.444302+0000 2024-09-17T15:51:58.446 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:58.446 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.446 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:58.446 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:58.447 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:58.447 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:58.447 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:58.447 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:58.447 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:58.447 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:58.448 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:58.448 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.444+0000 7ffb5be7c640 1 Processor -- start 2024-09-17T15:51:58.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.444+0000 7ffb5be7c640 1 -- start start 2024-09-17T15:51:58.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.444+0000 7ffb5be7c640 10 monclient: init 2024-09-17T15:51:58.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.444+0000 7ffb5be7c640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:58.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.444+0000 7ffb5be7c640 10 monclient: _add_conns ranks=[3,2,8,7,1,4,0,6,5] 2024-09-17T15:51:58.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.444+0000 7ffb5be7c640 10 monclient(hunting): picked mon.noname-g con 0x7ffb540a54b0 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:58.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.444+0000 7ffb5be7c640 10 monclient(hunting): picked mon.noname-b con 0x7ffb54135c80 addr v1:172.21.15.64:6790/0 2024-09-17T15:51:58.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.444+0000 7ffb5be7c640 10 monclient(hunting): picked mon.noname-e con 0x7ffb541360a0 addr v1:172.21.15.64:6793/0 2024-09-17T15:51:58.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.444+0000 7ffb5be7c640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb540aacb0 con 0x7ffb54135c80 2024-09-17T15:51:58.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.444+0000 7ffb5be7c640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb5409d780 con 0x7ffb540a54b0 2024-09-17T15:51:58.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.444+0000 7ffb5be7c640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb5409da60 con 0x7ffb541360a0 2024-09-17T15:51:58.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.444+0000 7ffb5be7c640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:58.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.444+0000 7ffb5be7c640 10 monclient(hunting): authenticate will time out at 1116.119982s 2024-09-17T15:51:58.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb59417640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7ffb54135c80 0x7ffb5413df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51512/0 (socket says 172.21.15.64:51512) 2024-09-17T15:51:58.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb59417640 1 -- 172.21.15.64:0/1219534401 learned_addr learned my addr 172.21.15.64:0/1219534401 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:58.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 -- 172.21.15.64:0/1219534401 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2774943277 0 0) 0x7ffb540aacb0 con 0x7ffb54135c80 2024-09-17T15:51:58.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient(hunting): my global_id is 124958 2024-09-17T15:51:58.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 -- 172.21.15.64:0/1219534401 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb40003520 con 0x7ffb54135c80 2024-09-17T15:51:58.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 -- 172.21.15.64:0/1219534401 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1652334410 0 0) 0x7ffb5409da60 con 0x7ffb541360a0 2024-09-17T15:51:58.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient(hunting): my global_id is 154241 2024-09-17T15:51:58.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 -- 172.21.15.64:0/1219534401 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb540aacb0 con 0x7ffb541360a0 2024-09-17T15:51:58.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 -- 172.21.15.64:0/1219534401 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2204875213 0 0) 0x7ffb5409d780 con 0x7ffb540a54b0 2024-09-17T15:51:58.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient(hunting): my global_id is 115608 2024-09-17T15:51:58.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 -- 172.21.15.64:0/1219534401 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb5409da60 con 0x7ffb540a54b0 2024-09-17T15:51:58.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 -- 172.21.15.64:0/1219534401 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1485300051 0 0) 0x7ffb40003520 con 0x7ffb54135c80 2024-09-17T15:51:58.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 -- 172.21.15.64:0/1219534401 >> v1:172.21.15.64:6793/0 conn(0x7ffb541360a0 legacy=0x7ffb54141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 -- 172.21.15.64:0/1219534401 >> v1:172.21.15.100:6790/0 conn(0x7ffb540a54b0 legacy=0x7ffb540a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient: _finish_hunting 0 2024-09-17T15:51:58.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 monclient: found mon.noname-b 2024-09-17T15:51:58.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:58.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:51:58.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 -- 172.21.15.64:0/1219534401 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb5409dd40 con 0x7ffb54135c80 2024-09-17T15:51:58.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient: _finish_auth 0 2024-09-17T15:51:58.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient: _check_auth_tickets 2024-09-17T15:51:58.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:58.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb5be7c640 5 monclient: authenticate success, global_id 124958 2024-09-17T15:51:58.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 -- 172.21.15.64:0/1219534401 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffb44001830 con 0x7ffb54135c80 2024-09-17T15:51:58.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:51:58.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient: dump: 2024-09-17T15:51:58.468 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.468 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.468 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.468 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.468 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.468 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.468 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.468 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.468 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.468 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.468 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.468 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.468 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb5be7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 -- 172.21.15.64:0/1219534401 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffb440019d0 con 0x7ffb54135c80 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 1 -- 172.21.15.64:0/1219534401 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffb44004be0 con 0x7ffb54135c80 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb5be7c640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb5be7c640 4 set_mon_vals no callback set 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.445+0000 7ffb58c16640 10 monclient: dump: 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.470 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.446+0000 7ffb5be7c640 10 monclient: shutdown 2024-09-17T15:51:58.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.446+0000 7ffb5be7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:58.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.446+0000 7ffb5be7c640 1 -- 172.21.15.64:0/1219534401 >> v1:172.21.15.64:6790/0 conn(0x7ffb54135c80 legacy=0x7ffb5413df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.446+0000 7ffb5be7c640 1 -- 172.21.15.64:0/1219534401 shutdown_connections 2024-09-17T15:51:58.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.446+0000 7ffb5be7c640 1 -- 172.21.15.64:0/1219534401 >> 172.21.15.64:0/1219534401 conn(0x7ffb5406daa0 msgr2=0x7ffb540ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:58.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.446+0000 7ffb5be7c640 1 -- 172.21.15.64:0/1219534401 shutdown_connections 2024-09-17T15:51:58.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.446+0000 7ffb5be7c640 1 -- 172.21.15.64:0/1219534401 wait complete. 2024-09-17T15:51:58.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.447+0000 7ffb5be7c640 10 monclient: build_initial_monmap 2024-09-17T15:51:58.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.447+0000 7ffb5be7c640 10 monclient: monmap: 2024-09-17T15:51:58.471 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:58.471 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:58.471 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.447+0000 7ffb5be7c640 1 Processor -- start 2024-09-17T15:51:58.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.447+0000 7ffb5be7c640 1 -- start start 2024-09-17T15:51:58.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.447+0000 7ffb5be7c640 10 monclient: init 2024-09-17T15:51:58.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb5be7c640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:58.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb5be7c640 10 monclient: _add_conns ranks=[5,7,1,0,6,3,8,2,4] 2024-09-17T15:51:58.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb5be7c640 10 monclient(hunting): picked mon.noname-h con 0x7ffb540a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:58.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb5be7c640 10 monclient(hunting): picked mon.noname-i con 0x7ffb54135c80 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:58.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb5be7c640 10 monclient(hunting): picked mon.noname-f con 0x7ffb541360a0 addr v1:172.21.15.100:6789/0 2024-09-17T15:51:58.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb5be7c640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb541cebf0 con 0x7ffb541360a0 2024-09-17T15:51:58.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb5be7c640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb540a3a50 con 0x7ffb540a54b0 2024-09-17T15:51:58.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb5be7c640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb540a72e0 con 0x7ffb54135c80 2024-09-17T15:51:58.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb5be7c640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:58.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb5be7c640 10 monclient(hunting): authenticate will time out at 1116.123197s 2024-09-17T15:51:58.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb5a419640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ffb541360a0 0x7ffb541d6030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.64:39712/0 (socket says 172.21.15.64:39712) 2024-09-17T15:51:58.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb5a419640 1 -- 172.21.15.64:0/943773178 learned_addr learned my addr 172.21.15.64:0/943773178 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:58.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb59c18640 1 --1- >> v1:172.21.15.100:6791/0 conn(0x7ffb540a54b0 0x7ffb541cddd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:47368/0 (socket says 172.21.15.64:47368) 2024-09-17T15:51:58.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1054479836 0 0) 0x7ffb541cebf0 con 0x7ffb541360a0 2024-09-17T15:51:58.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 10 monclient(hunting): my global_id is 6688 2024-09-17T15:51:58.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb38003520 con 0x7ffb541360a0 2024-09-17T15:51:58.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2728846233 0 0) 0x7ffb540a72e0 con 0x7ffb54135c80 2024-09-17T15:51:58.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 10 monclient(hunting): my global_id is 154294 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb541cebf0 con 0x7ffb54135c80 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3791609607 0 0) 0x7ffb540a3a50 con 0x7ffb540a54b0 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 10 monclient(hunting): my global_id is 154292 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb540a72e0 con 0x7ffb540a54b0 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2957668246 0 0) 0x7ffb541cebf0 con 0x7ffb54135c80 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.448+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb540a3a50 con 0x7ffb54135c80 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffb44004b20 con 0x7ffb54135c80 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient(hunting): dump: 2024-09-17T15:51:58.475 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.476 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.476 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.476 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.476 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.476 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.476 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.476 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.476 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.476 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.476 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.476 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.476 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.476 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 403475260 0 0) 0x7ffb540a72e0 con 0x7ffb540a54b0 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb541cebf0 con 0x7ffb540a54b0 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 723476171 0 0) 0x7ffb38003520 con 0x7ffb541360a0 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb540a72e0 con 0x7ffb541360a0 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffb50001550 con 0x7ffb540a54b0 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient(hunting): dump: 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.477 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.478 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.478 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.478 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.478 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.478 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.478 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.478 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.478 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.478 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.478 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.478 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.478 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.478 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffb4c001db0 con 0x7ffb541360a0 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient(hunting): dump: 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.479 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.480 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.480 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.480 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.480 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.480 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.480 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4129394241 0 0) 0x7ffb540a3a50 con 0x7ffb54135c80 2024-09-17T15:51:58.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 >> v1:172.21.15.100:6791/0 conn(0x7ffb540a54b0 legacy=0x7ffb541cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 >> v1:172.21.15.100:6789/0 conn(0x7ffb541360a0 legacy=0x7ffb541d6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:51:58.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 monclient: found mon.i 2024-09-17T15:51:58.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:58.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:51:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb540a6db0 con 0x7ffb54135c80 2024-09-17T15:51:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:51:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:51:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb5be7c640 5 monclient: authenticate success, global_id 154294 2024-09-17T15:51:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb5be7c640 10 monclient: _renew_subs 2024-09-17T15:51:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb5be7c640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:51:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb5be7c640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffb54135160 con 0x7ffb54135c80 2024-09-17T15:51:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb5be7c640 10 monclient: _renew_subs 2024-09-17T15:51:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb5be7c640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:51:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb5be7c640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7ffb540a3490 con 0x7ffb54135c80 2024-09-17T15:51:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.7 v1:172.21.15.100:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffb440011e0 con 0x7ffb54135c80 2024-09-17T15:51:58.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.7 v1:172.21.15.100:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffb440019a0 con 0x7ffb54135c80 2024-09-17T15:51:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4bfff640 4 set_mon_vals no callback set 2024-09-17T15:51:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:51:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb4a7fc640 10 monclient: dump: 2024-09-17T15:51:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.483 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.483 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.483 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.483 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.483 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.483 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.483 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.483 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.449+0000 7ffb5be7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:58.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.450+0000 7ffb5be7c640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ffb3c0065c0 con 0x7ffb3c002860 2024-09-17T15:51:58.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.450+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.7 v1:172.21.15.100:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7ffb440115e0 con 0x7ffb54135c80 2024-09-17T15:51:58.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.450+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.7 v1:172.21.15.100:6792/0 8 ==== osd_map(288..288 src has 192..288) ==== 3506+0+0 (unknown 2682836943 0 0) 0x7ffb4404e8f0 con 0x7ffb54135c80 2024-09-17T15:51:58.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.450+0000 7ffb5a419640 10 monclient: get_auth_request con 0x7ffb38040d70 auth_method 0 2024-09-17T15:51:58.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.450+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3756728396 0 0) 0x7ffb3c0065c0 con 0x7ffb3c002860 2024-09-17T15:51:58.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.450+0000 7ffb4a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.450+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.450+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.450+0000 7ffb4a7fc640 10 monclient(hunting): my global_id is 154301 2024-09-17T15:51:58.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.450+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb540a3a50 con 0x7ffb3c002860 2024-09-17T15:51:58.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.450+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2328215402 0 0) 0x7ffb540a3a50 con 0x7ffb3c002860 2024-09-17T15:51:58.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.451+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb3c0065c0 con 0x7ffb3c002860 2024-09-17T15:51:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.451+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffb50003930 con 0x7ffb3c002860 2024-09-17T15:51:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.451+0000 7ffb4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.451+0000 7ffb4a7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:51:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.451+0000 7ffb4a7fc640 10 monclient: dump: 2024-09-17T15:51:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.485 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.486 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.451+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2172692815 0 0) 0x7ffb3c0065c0 con 0x7ffb3c002860 2024-09-17T15:51:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.451+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffb3c00a160 con 0x7ffb3c002860 2024-09-17T15:51:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.451+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ffb50005780 con 0x7ffb3c002860 2024-09-17T15:51:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.451+0000 7ffb4a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.451+0000 7ffb4a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:58.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.451+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 >> v1:172.21.15.100:6791/0 conn(0x7ffb3c002860 legacy=0x7ffb3c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.451+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 >> v1:172.21.15.100:6791/0 conn(0x7ffb3c002860 legacy=0x7ffb3c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.457+0000 7ffb5be7c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.457+0000 7ffb5be7c640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ffb3c009e00 con 0x7ffb3c0090e0 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 616006704 0 0) 0x7ffb3c009e00 con 0x7ffb3c0090e0 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 10 monclient(hunting): my global_id is 154310 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb3c0065c0 con 0x7ffb3c0090e0 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1826000705 0 0) 0x7ffb3c0065c0 con 0x7ffb3c0090e0 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb3c009e00 con 0x7ffb3c0090e0 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7ffb50004230 con 0x7ffb3c0090e0 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 10 monclient: dump: 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.488 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.489 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.489 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.489 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3220382222 0 0) 0x7ffb3c009e00 con 0x7ffb3c0090e0 2024-09-17T15:51:58.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 --> v1:172.21.15.100:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffb3c00c3a0 con 0x7ffb3c0090e0 2024-09-17T15:51:58.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 1933581019) 0x7ffb50005930 con 0x7ffb3c0090e0 2024-09-17T15:51:58.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:58.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:58.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 >> v1:172.21.15.100:6791/0 conn(0x7ffb3c0090e0 legacy=0x7ffb3c009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.458+0000 7ffb4a7fc640 1 -- 172.21.15.64:0/943773178 >> v1:172.21.15.100:6791/0 conn(0x7ffb3c0090e0 legacy=0x7ffb3c009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:58.490 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:58.490 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:58.490 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:58.490 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:58.490 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:58.490 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 9800, 2024-09-17T15:51:58.490 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:58.490 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:58.490 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:58.490 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:58.490 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.491 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.492 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.492 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.492 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.492 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.492 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.492 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.492 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.492 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.492 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:58.492 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:58.492 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.492 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.492 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:58.493 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:58.494 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:58.494 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:58.494 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:58.494 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:58.494 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:58.494 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:58.494 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:58.494 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:58.494 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:58.494 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:58.494 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:58.494 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:58.494 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:58.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.496 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:58.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:58.496 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:58.496 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.496 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.496 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.496 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:58.496 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.496 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.496 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.497 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:58.497 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:58.497 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.497 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.497 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.497 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:58.497 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:58.497 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.497 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.497 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:58.498 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.499 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.500 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:58.500 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:58.500 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.500 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.500 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.500 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:58.500 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.500 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:58.502 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.504 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:58.504 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:58.504 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.504 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.504 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.504 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:58.504 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.504 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.504 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:58.505 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.506 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.506 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:58.506 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:58.506 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.506 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.506 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.506 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:58.506 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:58.506 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.507 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.507 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:58.507 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.507 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.507 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:58.507 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:58.507 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.507 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.507 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.508 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.508 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.508 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:58.508 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:58.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.508 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:58.508 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:58.508 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:58.508 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.508 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:58.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.509 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:58.509 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:58.509 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:58.509 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.509 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.509 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:58.509 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:58.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.459+0000 7ffb5be7c640 1 -- 172.21.15.64:0/943773178 >> v1:172.21.15.100:6812/4210963484 conn(0x7ffb38040d70 legacy=0x7ffb38043230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.459+0000 7ffb5be7c640 10 monclient: shutdown 2024-09-17T15:51:58.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.459+0000 7ffb5be7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:58.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.459+0000 7ffb5be7c640 1 -- 172.21.15.64:0/943773178 >> v1:172.21.15.100:6792/0 conn(0x7ffb54135c80 legacy=0x7ffb541ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.459+0000 7ffb5a419640 1 -- 172.21.15.64:0/943773178 reap_dead start 2024-09-17T15:51:58.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.459+0000 7ffb5a419640 1 -- 172.21.15.64:0/943773178 reap_dead start 2024-09-17T15:51:58.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.459+0000 7ffb5be7c640 1 -- 172.21.15.64:0/943773178 shutdown_connections 2024-09-17T15:51:58.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.459+0000 7ffb5be7c640 1 -- 172.21.15.64:0/943773178 >> 172.21.15.64:0/943773178 conn(0x7ffb5406daa0 msgr2=0x7ffb540a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:58.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.459+0000 7ffb5be7c640 1 -- 172.21.15.64:0/943773178 shutdown_connections 2024-09-17T15:51:58.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.459+0000 7ffb5be7c640 1 -- 172.21.15.64:0/943773178 wait complete. 2024-09-17T15:51:58.511 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:58.511 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-17T15:51:58.512 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-17T15:51:58.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.611+0000 7f6140d75640 10 monclient: get_monmap_and_config 2024-09-17T15:51:58.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.611+0000 7f6140d75640 10 monclient: build_initial_monmap 2024-09-17T15:51:58.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.611+0000 7f6140d75640 10 monclient: monmap: 2024-09-17T15:51:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:51:58.612665+0000 2024-09-17T15:51:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:51:58.612665+0000 2024-09-17T15:51:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:58.615 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:58.615 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:58.615 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.612+0000 7f6140d75640 1 Processor -- start 2024-09-17T15:51:58.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.612+0000 7f6140d75640 1 -- start start 2024-09-17T15:51:58.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.612+0000 7f6140d75640 10 monclient: init 2024-09-17T15:51:58.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6140d75640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:58.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6140d75640 10 monclient: _add_conns ranks=[5,8,0,7,2,3,6,4,1] 2024-09-17T15:51:58.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6140d75640 10 monclient(hunting): picked mon.noname-h con 0x7f613c0a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:58.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6140d75640 10 monclient(hunting): picked mon.noname-e con 0x7f613c135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:51:58.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6140d75640 10 monclient(hunting): picked mon.noname-a con 0x7f613c135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:51:58.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6140d75640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f613c09d6f0 con 0x7f613c135f70 2024-09-17T15:51:58.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6140d75640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f613c09d9d0 con 0x7f613c0a5520 2024-09-17T15:51:58.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6140d75640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f613c09dcb0 con 0x7f613c135b50 2024-09-17T15:51:58.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6140d75640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:58.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6140d75640 10 monclient(hunting): authenticate will time out at 1116.288197s 2024-09-17T15:51:58.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6139d9b640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f613c135b50 0x7f613c13deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:37128/0 (socket says 172.21.15.64:37128) 2024-09-17T15:51:58.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6139d9b640 1 -- 172.21.15.64:0/1943335384 learned_addr learned my addr 172.21.15.64:0/1943335384 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:58.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 1 -- 172.21.15.64:0/1943335384 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3992208367 0 0) 0x7f613c09d6f0 con 0x7f613c135f70 2024-09-17T15:51:58.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient(hunting): my global_id is 124875 2024-09-17T15:51:58.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 1 -- 172.21.15.64:0/1943335384 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6124003520 con 0x7f613c135f70 2024-09-17T15:51:58.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 1 -- 172.21.15.64:0/1943335384 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346986887 0 0) 0x7f613c09dcb0 con 0x7f613c135b50 2024-09-17T15:51:58.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient(hunting): my global_id is 154250 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 1 -- 172.21.15.64:0/1943335384 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f613c09d6f0 con 0x7f613c135b50 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 1 -- 172.21.15.64:0/1943335384 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 191816421 0 0) 0x7f613c09d9d0 con 0x7f613c0a5520 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient(hunting): my global_id is 154319 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 1 -- 172.21.15.64:0/1943335384 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f613c09dcb0 con 0x7f613c0a5520 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 1 -- 172.21.15.64:0/1943335384 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 333192090 0 0) 0x7f6124003520 con 0x7f613c135f70 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 1 -- 172.21.15.64:0/1943335384 >> v1:172.21.15.64:6793/0 conn(0x7f613c135b50 legacy=0x7f613c13deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 1 -- 172.21.15.64:0/1943335384 >> v1:172.21.15.100:6791/0 conn(0x7f613c0a5520 legacy=0x7f613c0a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient: _finish_hunting 0 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 1 monclient: found mon.noname-a 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:51:58.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 1 -- 172.21.15.64:0/1943335384 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613c09ccd0 con 0x7f613c135f70 2024-09-17T15:51:58.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient: _finish_auth 0 2024-09-17T15:51:58.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient: _check_auth_tickets 2024-09-17T15:51:58.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:58.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6140d75640 5 monclient: authenticate success, global_id 124875 2024-09-17T15:51:58.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 1 -- 172.21.15.64:0/1943335384 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6134001bb0 con 0x7f613c135f70 2024-09-17T15:51:58.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6140d75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:58.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:58.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 10 monclient: dump: 2024-09-17T15:51:58.626 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.626 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.626 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.627 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.627 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.627 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.627 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.627 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.627 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.627 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.627 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.627 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.627 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.627 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.627 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.627 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f6140d75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:58.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.613+0000 7f613959a640 1 -- 172.21.15.64:0/1943335384 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6134001d50 con 0x7f613c135f70 2024-09-17T15:51:58.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f613959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:58.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f613959a640 1 -- 172.21.15.64:0/1943335384 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6134003cd0 con 0x7f613c135f70 2024-09-17T15:51:58.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f6140d75640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:58.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f6140d75640 4 set_mon_vals no callback set 2024-09-17T15:51:58.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f613959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f613959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:58.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f613959a640 10 monclient: dump: 2024-09-17T15:51:58.628 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.628 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.628 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.628 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.629 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.629 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.629 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.629 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.629 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.629 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.629 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.629 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.629 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.629 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.629 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.629 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f6140d75640 10 monclient: shutdown 2024-09-17T15:51:58.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f6140d75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:58.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f6140d75640 1 -- 172.21.15.64:0/1943335384 >> v1:172.21.15.64:6789/0 conn(0x7f613c135f70 legacy=0x7f613c1415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f6140d75640 1 -- 172.21.15.64:0/1943335384 shutdown_connections 2024-09-17T15:51:58.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f6140d75640 1 -- 172.21.15.64:0/1943335384 >> 172.21.15.64:0/1943335384 conn(0x7f613c06db30 msgr2=0x7f613c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:58.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f6140d75640 1 -- 172.21.15.64:0/1943335384 shutdown_connections 2024-09-17T15:51:58.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.614+0000 7f6140d75640 1 -- 172.21.15.64:0/1943335384 wait complete. 2024-09-17T15:51:58.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.615+0000 7f6140d75640 10 monclient: build_initial_monmap 2024-09-17T15:51:58.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.615+0000 7f6140d75640 10 monclient: monmap: 2024-09-17T15:51:58.630 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:51:58.630 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:58.630 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:51:58.630 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:51:58.630 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.615+0000 7f6140d75640 1 Processor -- start 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.615+0000 7f6140d75640 1 -- start start 2024-09-17T15:51:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.615+0000 7f6140d75640 10 monclient: init 2024-09-17T15:51:58.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f6140d75640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:58.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f6140d75640 10 monclient: _add_conns ranks=[7,5,3,8,0,1,4,6,2] 2024-09-17T15:51:58.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f6140d75640 10 monclient(hunting): picked mon.noname-i con 0x7f613c0a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:51:58.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f6140d75640 10 monclient(hunting): picked mon.noname-h con 0x7f613c135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:51:58.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f6140d75640 10 monclient(hunting): picked mon.noname-g con 0x7f613c135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:51:58.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f6140d75640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f613c0a05b0 con 0x7f613c135f70 2024-09-17T15:51:58.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f6140d75640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f613c0a8f80 con 0x7f613c135b50 2024-09-17T15:51:58.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f6140d75640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f613c135300 con 0x7f613c0a5520 2024-09-17T15:51:58.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f6140d75640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:58.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f6140d75640 10 monclient(hunting): authenticate will time out at 1116.291226s 2024-09-17T15:51:58.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f613a59c640 1 --1- >> v1:172.21.15.100:6792/0 conn(0x7f613c0a5520 0x7f613c131720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6792/0 says I am v1:172.21.15.64:37494/0 (socket says 172.21.15.64:37494) 2024-09-17T15:51:58.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f613a59c640 1 -- 172.21.15.64:0/1393008389 learned_addr learned my addr 172.21.15.64:0/1393008389 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:51:58.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 457648976 0 0) 0x7f613c0a8f80 con 0x7f613c135b50 2024-09-17T15:51:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 10 monclient(hunting): my global_id is 154328 2024-09-17T15:51:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f611c003520 con 0x7f613c135b50 2024-09-17T15:51:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4014180617 0 0) 0x7f613c135300 con 0x7f613c0a5520 2024-09-17T15:51:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 10 monclient(hunting): my global_id is 154303 2024-09-17T15:51:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f613c0a8f80 con 0x7f613c0a5520 2024-09-17T15:51:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2758539740 0 0) 0x7f613c0a05b0 con 0x7f613c135f70 2024-09-17T15:51:58.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 10 monclient(hunting): my global_id is 115617 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.616+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f613c135300 con 0x7f613c135f70 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3120424335 0 0) 0x7f611c003520 con 0x7f613c135b50 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f613c0a05b0 con 0x7f613c135b50 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6130001610 con 0x7f613c135b50 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient(hunting): dump: 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:58.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2612289357 0 0) 0x7f613c0a8f80 con 0x7f613c0a5520 2024-09-17T15:51:58.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f611c003520 con 0x7f613c0a5520 2024-09-17T15:51:58.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f612c001610 con 0x7f613c0a5520 2024-09-17T15:51:58.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:58.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:51:58.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient(hunting): dump: 2024-09-17T15:51:58.636 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.636 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.636 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.636 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.636 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.636 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.636 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.636 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.637 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.637 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.637 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.637 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.637 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.637 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.637 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.637 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1326444193 0 0) 0x7f613c135300 con 0x7f613c135f70 2024-09-17T15:51:58.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f613c0a8f80 con 0x7f613c135f70 2024-09-17T15:51:58.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6134001af0 con 0x7f613c135f70 2024-09-17T15:51:58.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:51:58.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:51:58.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient(hunting): dump: 2024-09-17T15:51:58.638 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.638 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.638 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.638 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.638 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.638 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.638 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.638 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.638 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.638 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.638 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.638 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2695394326 0 0) 0x7f613c0a05b0 con 0x7f613c135b50 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 >> v1:172.21.15.100:6792/0 conn(0x7f613c0a5520 legacy=0x7f613c131720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 >> v1:172.21.15.100:6790/0 conn(0x7f613c135f70 legacy=0x7f613c1d5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient: _finish_hunting 0 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 monclient: found mon.h 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613c134db0 con 0x7f613c135b50 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient: _finish_auth 0 2024-09-17T15:51:58.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:58.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:58.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f6140d75640 5 monclient: authenticate success, global_id 154328 2024-09-17T15:51:58.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f6140d75640 10 monclient: _renew_subs 2024-09-17T15:51:58.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f6140d75640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:51:58.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f6140d75640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f613c132540 con 0x7f613c135b50 2024-09-17T15:51:58.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f6140d75640 10 monclient: _renew_subs 2024-09-17T15:51:58.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f6140d75640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.100:6791/0 2024-09-17T15:51:58.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f6140d75640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f613c09efa0 con 0x7f613c135b50 2024-09-17T15:51:58.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.5 v1:172.21.15.100:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6130004a90 con 0x7f613c135b50 2024-09-17T15:51:58.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:58.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.5 v1:172.21.15.100:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6130004f80 con 0x7f613c135b50 2024-09-17T15:51:58.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f6138d99640 4 set_mon_vals no callback set 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.617+0000 7f612affd640 10 monclient: dump: 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.641 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.642 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.642 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.642 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.642 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f6140d75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:58.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f6140d75640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f61200065c0 con 0x7f6120002860 2024-09-17T15:51:58.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.5 v1:172.21.15.100:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f61300118e0 con 0x7f613c135b50 2024-09-17T15:51:58.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.5 v1:172.21.15.100:6791/0 8 ==== osd_map(288..288 src has 192..288) ==== 3506+0+0 (unknown 2682836943 0 0) 0x7f613004ec50 con 0x7f613c135b50 2024-09-17T15:51:58.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2270948618 0 0) 0x7f61200065c0 con 0x7f6120002860 2024-09-17T15:51:58.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 10 monclient(hunting): my global_id is 154312 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f613c0a05b0 con 0x7f6120002860 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f613ad9d640 10 monclient: get_auth_request con 0x7f611c040c90 auth_method 0 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 266704165 0 0) 0x7f613c0a05b0 con 0x7f6120002860 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61200065c0 con 0x7f6120002860 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f612c003ba0 con 0x7f6120002860 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.618+0000 7f612affd640 10 monclient: dump: 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.643 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.644 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.644 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.644 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.644 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.644 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.644 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.644 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.644 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.644 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.644 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.619+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 690515998 0 0) 0x7f61200065c0 con 0x7f6120002860 2024-09-17T15:51:58.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.619+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f612000a160 con 0x7f6120002860 2024-09-17T15:51:58.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.619+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f612c003e50 con 0x7f6120002860 2024-09-17T15:51:58.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.619+0000 7f612affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:58.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.619+0000 7f612affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:58.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.619+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 >> v1:172.21.15.100:6792/0 conn(0x7f6120002860 legacy=0x7f6120004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.619+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 >> v1:172.21.15.100:6792/0 conn(0x7f6120002860 legacy=0x7f6120004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:58.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.625+0000 7f6140d75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:58.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.625+0000 7f6140d75640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6120009e50 con 0x7f6120009110 2024-09-17T15:51:58.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3153538500 0 0) 0x7f6120009e50 con 0x7f6120009110 2024-09-17T15:51:58.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 10 monclient(hunting): _negotiate 2024-09-17T15:51:58.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:58.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 10 monclient(hunting): my global_id is 154321 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61200065c0 con 0x7f6120009110 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2004779859 0 0) 0x7f61200065c0 con 0x7f6120009110 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6120009e50 con 0x7f6120009110 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f612c005780 con 0x7f6120009110 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 10 monclient: dump: 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:51:58.648 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:51:58.648 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:51:58.648 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:51:58.648 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:51:58.648 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:51:58.648 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:51:58.648 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:51:58.648 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:51:58.648 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:51:58.648 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:51:58.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4008643138 0 0) 0x7f6120009e50 con 0x7f6120009110 2024-09-17T15:51:58.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 --> v1:172.21.15.100:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f612000c3f0 con 0x7f6120009110 2024-09-17T15:51:58.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 2436929550) 0x7f612c007980 con 0x7f6120009110 2024-09-17T15:51:58.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:58.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:58.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 >> v1:172.21.15.100:6792/0 conn(0x7f6120009110 legacy=0x7f61200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.626+0000 7f612affd640 1 -- 172.21.15.64:0/1393008389 >> v1:172.21.15.100:6792/0 conn(0x7f6120009110 legacy=0x7f61200095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:58.649 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:51:58.649 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:58.649 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:58.649 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:51:58.649 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 32, 2024-09-17T15:51:58.649 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 9973, 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 4, 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:58.650 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:58.651 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:51:58.651 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.651 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.651 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.651 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.651 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.651 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.651 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.651 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.651 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.651 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.651 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.651 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.652 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:58.652 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:51:58.652 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.652 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.652 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.652 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.652 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.652 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.652 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.652 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.652 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.652 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.652 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:51:58.653 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:58.654 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:51:58.654 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:58.654 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:51:58.654 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:51:58.654 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:51:58.654 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:51:58.654 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:51:58.654 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:51:58.656 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:51:58.656 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:51:58.656 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:51:58.656 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:51:58.656 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:51:58.656 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:51:58.656 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:51:58.656 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:51:58.657 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.657 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:51:58.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.657 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:51:58.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.657 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:51:58.657 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:51:58.657 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.657 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.657 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.657 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:51:58.657 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.658 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.658 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:51:58.658 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:51:58.658 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.658 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.658 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.658 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:51:58.658 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:51:58.658 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.660 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:51:58.660 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:51:58.660 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.660 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.660 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.660 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:51:58.660 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.660 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:51:58.661 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:51:58.661 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.661 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.661 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.661 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:51:58.661 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:51:58.661 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.661 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.661 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.661 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:51:58.662 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.663 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.663 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:51:58.663 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.663 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.663 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:51:58.663 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:51:58.663 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.663 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.663 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.664 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:51:58.664 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:51:58.664 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.664 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.664 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.664 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:51:58.664 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.664 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.664 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.665 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:51:58.665 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:51:58.665 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.665 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.665 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.665 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:51:58.665 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:51:58.665 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.665 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.665 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:51:58.666 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.667 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.668 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:51:58.668 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:51:58.668 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:51:58.668 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:51:58.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.668 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:51:58.668 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:51:58.668 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:51:58.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.668 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.669 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:51:58.669 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:51:58.669 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:51:58.669 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:51:58.669 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:51:58.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.669 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.669 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:51:58.669 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:51:58.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.669 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:58.669 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:58.670 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:51:58.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.670 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:51:58.670 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:51:58.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:51:58.670 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:58.670 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:51:58.670 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:51:58.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:51:58.670 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:51:58.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:51:58.670 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:51:58.670 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:51:58.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.627+0000 7f6140d75640 1 -- 172.21.15.64:0/1393008389 >> v1:172.21.15.100:6812/4210963484 conn(0x7f611c040c90 legacy=0x7f611c043150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.627+0000 7f6140d75640 10 monclient: shutdown 2024-09-17T15:51:58.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.627+0000 7f6140d75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:58.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.627+0000 7f6140d75640 1 -- 172.21.15.64:0/1393008389 >> v1:172.21.15.100:6791/0 conn(0x7f613c135b50 legacy=0x7f613c131e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:58.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.627+0000 7f613ad9d640 1 -- 172.21.15.64:0/1393008389 reap_dead start 2024-09-17T15:51:58.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.627+0000 7f613ad9d640 1 -- 172.21.15.64:0/1393008389 reap_dead start 2024-09-17T15:51:58.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.627+0000 7f6140d75640 1 -- 172.21.15.64:0/1393008389 shutdown_connections 2024-09-17T15:51:58.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.627+0000 7f6140d75640 1 -- 172.21.15.64:0/1393008389 >> 172.21.15.64:0/1393008389 conn(0x7f613c06db30 msgr2=0x7f613c1334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:58.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.627+0000 7f6140d75640 1 -- 172.21.15.64:0/1393008389 shutdown_connections 2024-09-17T15:51:58.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:51:58.627+0000 7f6140d75640 1 -- 172.21.15.64:0/1393008389 wait complete. 2024-09-17T15:51:58.673 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:51:58.673 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['d', 'i', 'e', 'h'] 2024-09-17T15:51:58.673 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-17T15:51:58.673 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-09-17T15:51:58.673 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-09-17T15:51:58.674 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-09-17T15:51:58.674 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:51:58.684 INFO:tasks.ceph.mon.d:Stopped 2024-09-17T15:51:58.684 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i 2024-09-17T15:51:58.684 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-09-17T15:51:58.684 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-09-17T15:51:58.684 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:51:58.696 INFO:tasks.ceph.mon.i:Stopped 2024-09-17T15:51:58.696 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-09-17T15:51:58.696 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-09-17T15:51:58.696 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-09-17T15:51:58.696 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:51:58.734 INFO:tasks.ceph.mon.e:Stopped 2024-09-17T15:51:58.734 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h 2024-09-17T15:51:58.734 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-09-17T15:51:58.734 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-09-17T15:51:58.735 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:51:58.746 INFO:tasks.ceph.mon.h:Stopped 2024-09-17T15:51:58.746 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-17T15:51:58.747 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-17T15:51:58.747 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T15:52:13.747 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-17T15:52:13.748 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-17T15:52:13.748 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-09-17T15:52:13.749 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:52:13.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.858+0000 7f5406a31640 10 monclient: get_monmap_and_config 2024-09-17T15:52:13.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.858+0000 7f5406a31640 10 monclient: build_initial_monmap 2024-09-17T15:52:13.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.859+0000 7f5406a31640 10 monclient: monmap: 2024-09-17T15:52:13.862 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:52:13.862 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:13.862 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:52:13.860854+0000 2024-09-17T15:52:13.862 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:52:13.860854+0000 2024-09-17T15:52:13.862 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:13.863 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:13.863 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:52:13.863 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:52:13.863 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:52:13.863 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:52:13.863 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:52:13.863 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:52:13.863 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:52:13.863 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:52:13.863 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:52:13.863 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:13.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.859+0000 7f5406a31640 1 Processor -- start 2024-09-17T15:52:13.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.859+0000 7f5406a31640 1 -- start start 2024-09-17T15:52:13.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.859+0000 7f5406a31640 10 monclient: init 2024-09-17T15:52:13.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f5406a31640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:13.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f5406a31640 10 monclient: _add_conns ranks=[8,2,0,5,6,3,1,7,4] 2024-09-17T15:52:13.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f5406a31640 10 monclient(hunting): picked mon.noname-e con 0x7f54000a54b0 addr v1:172.21.15.64:6793/0 2024-09-17T15:52:13.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f5406a31640 10 monclient(hunting): picked mon.noname-b con 0x7f5400135c80 addr v1:172.21.15.64:6790/0 2024-09-17T15:52:13.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f5406a31640 10 monclient(hunting): picked mon.noname-a con 0x7f54001360a0 addr v1:172.21.15.64:6789/0 2024-09-17T15:52:13.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f5406a31640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54000aacb0 con 0x7f54001360a0 2024-09-17T15:52:13.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f5406a31640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540009d780 con 0x7f5400135c80 2024-09-17T15:52:13.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f5406a31640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540009da60 con 0x7f54000a54b0 2024-09-17T15:52:13.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f5406a31640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:13.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f5406a31640 10 monclient(hunting): authenticate will time out at 1131.536496s 2024-09-17T15:52:13.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f53fffff640 1 -- >> v1:172.21.15.64:6793/0 conn(0x7f54000a54b0 legacy=0x7f54000a5890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:52:13.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f5404fce640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f54001360a0 0x7f5400141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:52088/0 (socket says 172.21.15.64:52088) 2024-09-17T15:52:13.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 learned_addr learned my addr 172.21.15.64:0/2713073860 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:52:13.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:13.860+0000 7f53ff7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f5400135c80 0x7f540013df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:59186/0 (socket says 172.21.15.64:59186) 2024-09-17T15:52:14.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:14.061+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6793/0 conn(0x7f54000a54b0 legacy=0x7f54000a5890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:52:14.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:14.462+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6793/0 conn(0x7f54000a54b0 legacy=0x7f54000a5890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:52:15.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:15.263+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6793/0 conn(0x7f54000a54b0 legacy=0x7f54000a5890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:52:16.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 10 monclient(hunting): tick 2024-09-17T15:52:16.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:52:16.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 1 monclient(hunting): continuing hunt 2024-09-17T15:52:16.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:52:16.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6793/0 conn(0x7f54000a54b0 legacy=0x7f54000a5890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:16.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6790/0 conn(0x7f5400135c80 legacy=0x7f540013df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:16.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6789/0 conn(0x7f54001360a0 legacy=0x7f5400141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:16.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 10 monclient: _add_conns ranks=[5,4,7,3,6,1] 2024-09-17T15:52:16.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 10 monclient(hunting): picked mon.noname-h con 0x7f53e00037c0 addr v1:172.21.15.100:6791/0 2024-09-17T15:52:16.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 10 monclient(hunting): picked mon.noname-c con 0x7f53e0007450 addr v1:172.21.15.64:6791/0 2024-09-17T15:52:16.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 10 monclient(hunting): picked mon.noname-i con 0x7f53e000b1e0 addr v1:172.21.15.100:6792/0 2024-09-17T15:52:16.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540009dd40 con 0x7f53e0007450 2024-09-17T15:52:16.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540009da60 con 0x7f53e00037c0 2024-09-17T15:52:16.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e0011010 con 0x7f53e000b1e0 2024-09-17T15:52:16.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.861+0000 7f53fdffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:16.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.862+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6791/0 conn(0x7f53e00037c0 legacy=0x7f53e0005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:52:16.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:16.862+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6792/0 conn(0x7f53e000b1e0 legacy=0x7f53e000d680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:52:17.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:17.063+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6791/0 conn(0x7f53e00037c0 legacy=0x7f53e0005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:52:17.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:17.063+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6792/0 conn(0x7f53e000b1e0 legacy=0x7f53e000d680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:52:17.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:17.464+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6791/0 conn(0x7f53e00037c0 legacy=0x7f53e0005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:52:17.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:17.464+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6792/0 conn(0x7f53e000b1e0 legacy=0x7f53e000d680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:52:18.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:18.265+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6792/0 conn(0x7f53e000b1e0 legacy=0x7f53e000d680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:52:18.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:18.265+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6791/0 conn(0x7f53e00037c0 legacy=0x7f53e0005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:52:19.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 10 monclient(hunting): tick 2024-09-17T15:52:19.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:52:19.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 1 monclient(hunting): continuing hunt 2024-09-17T15:52:19.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:52:19.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6792/0 conn(0x7f53e000b1e0 legacy=0x7f53e000d680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:19.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6791/0 conn(0x7f53e00037c0 legacy=0x7f53e0005c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:19.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6791/0 conn(0x7f53e0007450 legacy=0x7f53e0009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 reap_dead start 2024-09-17T15:52:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 10 monclient: _add_conns ranks=[3,1,6] 2024-09-17T15:52:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 10 monclient(hunting): picked mon.noname-g con 0x7f53e00112e0 addr v1:172.21.15.100:6790/0 2024-09-17T15:52:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 10 monclient(hunting): picked mon.noname-f con 0x7f53e000de50 addr v1:172.21.15.100:6789/0 2024-09-17T15:52:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 10 monclient(hunting): picked mon.noname-d con 0x7f53e000e2e0 addr v1:172.21.15.64:6792/0 2024-09-17T15:52:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e0012310 con 0x7f53e000de50 2024-09-17T15:52:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540009da60 con 0x7f53e00112e0 2024-09-17T15:52:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e0011010 con 0x7f53e000e2e0 2024-09-17T15:52:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f53fdffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:19.862+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6792/0 conn(0x7f53e000e2e0 legacy=0x7f53e000e790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:52:20.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:20.063+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6792/0 conn(0x7f53e000e2e0 legacy=0x7f53e000e790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:52:20.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:20.464+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6792/0 conn(0x7f53e000e2e0 legacy=0x7f53e000e790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:52:21.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:21.265+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6792/0 conn(0x7f53e000e2e0 legacy=0x7f53e000e790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:52:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:21.807+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6790/0 conn(0x7f53e00112e0 legacy=0x7f53e00116c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-17T15:52:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:21.807+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6790/0 conn(0x7f53e00112e0 legacy=0x7f53e00116c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:52:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:21.807+0000 7f53fffff640 1 --1- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6790/0 conn(0x7f53e00112e0 0x7f53e00116c0 :-1 s=OPENED pgs=419 cs=1 l=1).handle_message read tag failed 2024-09-17T15:52:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:21.807+0000 7f53fffff640 1 --1- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6790/0 conn(0x7f53e00112e0 0x7f53e00116c0 :-1 s=OPENED pgs=419 cs=1 l=1).fault on lossy channel, failing 2024-09-17T15:52:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:21.807+0000 7f53feffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.100:6790/0 2024-09-17T15:52:22.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 10 monclient(hunting): tick 2024-09-17T15:52:22.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:52:22.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 1 monclient(hunting): continuing hunt 2024-09-17T15:52:22.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:52:22.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6792/0 conn(0x7f53e000e2e0 legacy=0x7f53e000e790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:22.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6790/0 conn(0x7f53e00112e0 legacy=0x7f53e00116c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:22.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6789/0 conn(0x7f53e000de50 legacy=0x7f53e000d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:22.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 10 monclient: _add_conns ranks=[7,1,2,0,3,5,6,8,4] 2024-09-17T15:52:22.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 10 monclient(hunting): picked mon.noname-i con 0x7f53e0005f10 addr v1:172.21.15.100:6792/0 2024-09-17T15:52:22.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 10 monclient(hunting): picked mon.noname-f con 0x7f53e000a180 addr v1:172.21.15.100:6789/0 2024-09-17T15:52:22.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 10 monclient(hunting): picked mon.noname-b con 0x7f53e0018680 addr v1:172.21.15.64:6790/0 2024-09-17T15:52:22.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e0005c60 con 0x7f53e000a180 2024-09-17T15:52:22.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e0011010 con 0x7f53e0018680 2024-09-17T15:52:22.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e000c680 con 0x7f53e0005f10 2024-09-17T15:52:22.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fdffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:22.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:22.862+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6792/0 conn(0x7f53e0005f10 legacy=0x7f53e0009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:52:23.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:23.063+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6792/0 conn(0x7f53e0005f10 legacy=0x7f53e0009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:52:23.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:23.101+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6790/0 conn(0x7f53e0018680 legacy=0x7f53e001aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-17T15:52:23.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:23.101+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6790/0 conn(0x7f53e0018680 legacy=0x7f53e001aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:52:23.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:23.101+0000 7f5404fce640 1 --1- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6790/0 conn(0x7f53e0018680 0x7f53e001aa70 :-1 s=OPENED pgs=366 cs=1 l=1).handle_message read tag failed 2024-09-17T15:52:23.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:23.101+0000 7f5404fce640 1 --1- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6790/0 conn(0x7f53e0018680 0x7f53e001aa70 :-1 s=OPENED pgs=366 cs=1 l=1).fault on lossy channel, failing 2024-09-17T15:52:23.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:23.101+0000 7f53feffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.64:6790/0 2024-09-17T15:52:23.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:23.431+0000 7f53ff7fe640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6789/0 conn(0x7f53e000a180 legacy=0x7f53e000a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-17T15:52:23.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:23.431+0000 7f53ff7fe640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6789/0 conn(0x7f53e000a180 legacy=0x7f53e000a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:52:23.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:23.431+0000 7f53ff7fe640 1 --1- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6789/0 conn(0x7f53e000a180 0x7f53e000a5c0 :-1 s=OPENED pgs=621 cs=1 l=1).handle_message read tag failed 2024-09-17T15:52:23.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:23.431+0000 7f53ff7fe640 1 --1- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6789/0 conn(0x7f53e000a180 0x7f53e000a5c0 :-1 s=OPENED pgs=621 cs=1 l=1).fault on lossy channel, failing 2024-09-17T15:52:23.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:23.431+0000 7f53feffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.100:6789/0 2024-09-17T15:52:23.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:23.464+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6792/0 conn(0x7f53e0005f10 legacy=0x7f53e0009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:52:24.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:24.266+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6792/0 conn(0x7f53e0005f10 legacy=0x7f53e0009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:52:25.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f53fdffb640 10 monclient(hunting): tick 2024-09-17T15:52:25.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f53fdffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:52:25.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f53fdffb640 1 monclient(hunting): continuing hunt 2024-09-17T15:52:25.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f53fdffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:52:25.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6792/0 conn(0x7f53e0005f10 legacy=0x7f53e0009890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:25.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6790/0 conn(0x7f53e0018680 legacy=0x7f53e001aa70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:25.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.100:6789/0 conn(0x7f53e000a180 legacy=0x7f53e000a5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:25.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f53fdffb640 10 monclient: _add_conns ranks=[6,4,8,3,5,0] 2024-09-17T15:52:25.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 reap_dead start 2024-09-17T15:52:25.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f53fdffb640 10 monclient(hunting): picked mon.noname-d con 0x7f53e000e2e0 addr v1:172.21.15.64:6792/0 2024-09-17T15:52:25.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f53fdffb640 10 monclient(hunting): picked mon.noname-c con 0x7f53e000b670 addr v1:172.21.15.64:6791/0 2024-09-17T15:52:25.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f53fdffb640 10 monclient(hunting): picked mon.noname-e con 0x7f53e0005f10 addr v1:172.21.15.64:6793/0 2024-09-17T15:52:25.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e000d860 con 0x7f53e000b670 2024-09-17T15:52:25.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.862+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e000c680 con 0x7f53e000e2e0 2024-09-17T15:52:25.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53fdffb640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e000db80 con 0x7f53e0005f10 2024-09-17T15:52:25.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53fdffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:25.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53fffff640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6792/0 conn(0x7f53e000e2e0 legacy=0x7f53e000e790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:52:25.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f5404fce640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6793/0 conn(0x7f53e0005f10 legacy=0x7f53e000a5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:52:25.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 1 -- 172.21.15.64:0/2713073860 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2051681138 0 0) 0x7f53e000d860 con 0x7f53e000b670 2024-09-17T15:52:25.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 10 monclient(hunting): _negotiate 2024-09-17T15:52:25.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:25.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:25.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 10 monclient(hunting): my global_id is 154273 2024-09-17T15:52:25.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53ec003520 con 0x7f53e000b670 2024-09-17T15:52:25.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 1 -- 172.21.15.64:0/2713073860 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2892307990 0 0) 0x7f53ec003520 con 0x7f53e000b670 2024-09-17T15:52:25.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6793/0 conn(0x7f53e0005f10 legacy=0x7f53e000a5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:25.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6792/0 conn(0x7f53e000e2e0 legacy=0x7f53e000e790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:25.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 10 monclient: _finish_hunting 0 2024-09-17T15:52:25.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 1 monclient: found mon.noname-c 2024-09-17T15:52:25.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 1 -- 172.21.15.64:0/2713073860 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e000c200 con 0x7f53e000b670 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 10 monclient: _finish_auth 0 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 10 monclient: _check_auth_tickets 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 1 -- 172.21.15.64:0/2713073860 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f53e80016f0 con 0x7f53e000b670 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f5406a31640 5 monclient: authenticate success, global_id 154273 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f5406a31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f53feffd640 10 monclient: dump: 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:25.869 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:25.870 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:25.870 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:25.870 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:25.870 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:25.870 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:25.870 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:25.870 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:25.870 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:25.870 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.863+0000 7f5406a31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.864+0000 7f53feffd640 1 -- 172.21.15.64:0/2713073860 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53e80018b0 con 0x7f53e000b670 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.864+0000 7f53feffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.864+0000 7f53feffd640 1 -- 172.21.15.64:0/2713073860 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f53e80047e0 con 0x7f53e000b670 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.864+0000 7f5406a31640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.864+0000 7f5406a31640 4 set_mon_vals no callback set 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.864+0000 7f53feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.864+0000 7f53feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.864+0000 7f53feffd640 10 monclient: dump: 2024-09-17T15:52:25.871 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:25.872 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:25.872 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:25.872 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:25.872 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:25.872 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:25.872 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:25.872 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:25.872 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:25.872 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:25.872 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:25.872 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:25.872 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:25.872 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:25.873 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:25.873 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:25.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.864+0000 7f5406a31640 10 monclient: shutdown 2024-09-17T15:52:25.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.864+0000 7f5406a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:25.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.864+0000 7f5406a31640 1 -- 172.21.15.64:0/2713073860 >> v1:172.21.15.64:6791/0 conn(0x7f53e000b670 legacy=0x7f53e000ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:25.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.864+0000 7f5406a31640 1 -- 172.21.15.64:0/2713073860 shutdown_connections 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.864+0000 7f5406a31640 1 -- 172.21.15.64:0/2713073860 >> 172.21.15.64:0/2713073860 conn(0x7f540006daa0 msgr2=0x7f54000ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.865+0000 7f5406a31640 1 -- 172.21.15.64:0/2713073860 shutdown_connections 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.865+0000 7f5406a31640 1 -- 172.21.15.64:0/2713073860 wait complete. 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.865+0000 7f5406a31640 10 monclient: build_initial_monmap 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.865+0000 7f5406a31640 10 monclient: monmap: 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:52:25.878 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.866+0000 7f5406a31640 1 Processor -- start 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.866+0000 7f5406a31640 1 -- start start 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.866+0000 7f5406a31640 10 monclient: init 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.866+0000 7f5406a31640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.866+0000 7f5406a31640 10 monclient: _add_conns ranks=[4,3,1,2,0,6,5,8,7] 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.866+0000 7f5406a31640 10 monclient(hunting): picked mon.noname-c con 0x7f53e0005f10 addr v1:172.21.15.64:6791/0 2024-09-17T15:52:25.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f5406a31640 10 monclient(hunting): picked mon.noname-g con 0x7f53e000b670 addr v1:172.21.15.100:6790/0 2024-09-17T15:52:25.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f5406a31640 10 monclient(hunting): picked mon.noname-f con 0x7f53e000e2e0 addr v1:172.21.15.100:6789/0 2024-09-17T15:52:25.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f5406a31640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54000a26d0 con 0x7f53e000e2e0 2024-09-17T15:52:25.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f5406a31640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54000a29b0 con 0x7f53e000b670 2024-09-17T15:52:25.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f5406a31640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54001ce6c0 con 0x7f53e0005f10 2024-09-17T15:52:25.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f5406a31640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:25.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f5406a31640 10 monclient(hunting): authenticate will time out at 1143.542094s 2024-09-17T15:52:25.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fffff640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f53e0005f10 0x7f54001cd8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:39822/0 (socket says 172.21.15.64:39822) 2024-09-17T15:52:25.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fffff640 1 -- 172.21.15.64:0/3892451406 learned_addr learned my addr 172.21.15.64:0/3892451406 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:52:25.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 177572061 0 0) 0x7f54001ce6c0 con 0x7f53e0005f10 2024-09-17T15:52:25.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): _negotiate 2024-09-17T15:52:25.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:25.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:25.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): my global_id is 154282 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53e000db60 con 0x7f53e0005f10 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 648159442 0 0) 0x7f53e000db60 con 0x7f53e0005f10 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54001ce6c0 con 0x7f53e0005f10 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3081714540 0 0) 0x7f54000a26d0 con 0x7f53e000e2e0 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): _negotiate 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): my global_id is 6697 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53e000db60 con 0x7f53e000e2e0 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f53f0004490 con 0x7f53e0005f10 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): dump: 2024-09-17T15:52:25.881 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:25.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1757068846 0 0) 0x7f54000a29b0 con 0x7f53e000b670 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): _negotiate 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 10 monclient(hunting): my global_id is 115626 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.867+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54000a26d0 con 0x7f53e000b670 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2665203438 0 0) 0x7f54001ce6c0 con 0x7f53e0005f10 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 >> v1:172.21.15.100:6790/0 conn(0x7f53e000b670 legacy=0x7f54001cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 >> v1:172.21.15.100:6789/0 conn(0x7f53e000e2e0 legacy=0x7f54001d5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 10 monclient: _finish_hunting 0 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 1 monclient: found mon.c 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54000a1b40 con 0x7f53e0005f10 2024-09-17T15:52:25.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 10 monclient: _finish_auth 0 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 10 monclient: _check_auth_tickets 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f5406a31640 5 monclient: authenticate success, global_id 154282 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f5406a31640 10 monclient: _renew_subs 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f5406a31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f5406a31640 1 -- 172.21.15.64:0/3892451406 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f54000a0650 con 0x7f53e0005f10 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f5406a31640 10 monclient: _renew_subs 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f5406a31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f5406a31640 1 -- 172.21.15.64:0/3892451406 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f54000a2dd0 con 0x7f53e0005f10 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53f00049d0 con 0x7f53e0005f10 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f53f0004f40 con 0x7f53e0005f10 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:25.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fe7fc640 4 set_mon_vals no callback set 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.868+0000 7f53fcff9640 10 monclient: dump: 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:25.885 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.869+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f53f00050e0 con 0x7f53e0005f10 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.869+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(290..290 src has 192..290) ==== 3898+0+0 (unknown 2326613063 0 0) 0x7f53f004ec70 con 0x7f53e0005f10 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.869+0000 7f5406a31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.869+0000 7f5406a31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.869+0000 7f5406a31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.869+0000 7f5406a31640 1 -- 172.21.15.64:0/3892451406 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53e4005080 con 0x7f53e0005f10 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.869+0000 7f53ff7fe640 10 monclient: get_auth_request con 0x7f53e0043e40 auth_method 0 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.874+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f53f00178e0 con 0x7f53e0005f10 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.875+0000 7f53fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:25.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:25.875+0000 7f53fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.140+0000 7f5406a31640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:52:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.140+0000 7f5406a31640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:52:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.140+0000 7f5406a31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.140+0000 7f5406a31640 1 -- 172.21.15.64:0/3892451406 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f53e4005370 con 0x7f53e0005f10 2024-09-17T15:52:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.140+0000 7f53fcff9640 1 -- 172.21.15.64:0/3892451406 <== mon.4 v1:172.21.15.64:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2705 (unknown 254746657 0 4159385386) 0x7f53f00171e0 con 0x7f53e0005f10 2024-09-17T15:52:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.140+0000 7f53fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:52:26.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.140+0000 7f53fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:26.143 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":40,"quorum":[0,1,2,3,4],"quorum_names":["a","f","b","g","c"],"quorum_leader_name":"f","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:52:26.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.143+0000 7f5406a31640 1 -- 172.21.15.64:0/3892451406 >> v1:172.21.15.100:6812/4210963484 conn(0x7f53e0043e40 legacy=0x7f53e0046300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.143+0000 7f5406a31640 10 monclient: shutdown 2024-09-17T15:52:26.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.143+0000 7f5406a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.143+0000 7f5406a31640 1 -- 172.21.15.64:0/3892451406 >> v1:172.21.15.64:6791/0 conn(0x7f53e0005f10 legacy=0x7f54001cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.143+0000 7f5406a31640 1 -- 172.21.15.64:0/3892451406 shutdown_connections 2024-09-17T15:52:26.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.143+0000 7f5406a31640 1 -- 172.21.15.64:0/3892451406 >> 172.21.15.64:0/3892451406 conn(0x7f540006daa0 msgr2=0x7f54000a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.143+0000 7f5406a31640 1 -- 172.21.15.64:0/3892451406 shutdown_connections 2024-09-17T15:52:26.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.143+0000 7f5406a31640 1 -- 172.21.15.64:0/3892451406 wait complete. 2024-09-17T15:52:26.155 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-09-17T15:52:26.156 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon a 2024-09-17T15:52:26.156 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:52:26.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.256+0000 7fb1e65d0640 10 monclient: get_monmap_and_config 2024-09-17T15:52:26.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.256+0000 7fb1e65d0640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.256+0000 7fb1e65d0640 10 monclient: monmap: 2024-09-17T15:52:26.258 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:52:26.258 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.258 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:52:26.256993+0000 2024-09-17T15:52:26.258 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:52:26.256993+0000 2024-09-17T15:52:26.258 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.258 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.258 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:52:26.259 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:52:26.259 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:52:26.259 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:52:26.259 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:52:26.259 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:52:26.259 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:52:26.259 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:52:26.259 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:52:26.259 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.256+0000 7fb1e65d0640 1 Processor -- start 2024-09-17T15:52:26.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.256+0000 7fb1e65d0640 1 -- start start 2024-09-17T15:52:26.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.256+0000 7fb1e65d0640 10 monclient: init 2024-09-17T15:52:26.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1e65d0640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1e65d0640 10 monclient: _add_conns ranks=[4,2,3,0,8,7,1,6,5] 2024-09-17T15:52:26.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1e65d0640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e00a54b0 addr v1:172.21.15.64:6791/0 2024-09-17T15:52:26.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1e65d0640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e0135c80 addr v1:172.21.15.64:6790/0 2024-09-17T15:52:26.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1e65d0640 10 monclient(hunting): picked mon.noname-g con 0x7fb1e01360a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:52:26.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1e65d0640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e00aacb0 con 0x7fb1e0135c80 2024-09-17T15:52:26.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1e65d0640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e009d780 con 0x7fb1e01360a0 2024-09-17T15:52:26.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1e65d0640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e009da60 con 0x7fb1e00a54b0 2024-09-17T15:52:26.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1e65d0640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1e65d0640 10 monclient(hunting): authenticate will time out at 1143.932471s 2024-09-17T15:52:26.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1dffff640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fb1e00a54b0 0x7fb1e00a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:39830/0 (socket says 172.21.15.64:39830) 2024-09-17T15:52:26.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1dffff640 1 -- 172.21.15.64:0/181412683 learned_addr learned my addr 172.21.15.64:0/181412683 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:52:26.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1deffd640 1 -- 172.21.15.64:0/181412683 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3703058451 0 0) 0x7fb1e009da60 con 0x7fb1e00a54b0 2024-09-17T15:52:26.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1deffd640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1deffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1deffd640 10 monclient(hunting): my global_id is 154291 2024-09-17T15:52:26.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1deffd640 1 -- 172.21.15.64:0/181412683 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1c8003520 con 0x7fb1e00a54b0 2024-09-17T15:52:26.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1deffd640 1 -- 172.21.15.64:0/181412683 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3315677011 0 0) 0x7fb1e00aacb0 con 0x7fb1e0135c80 2024-09-17T15:52:26.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1deffd640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1deffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1deffd640 10 monclient(hunting): my global_id is 124967 2024-09-17T15:52:26.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.257+0000 7fb1deffd640 1 -- 172.21.15.64:0/181412683 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1e009da60 con 0x7fb1e0135c80 2024-09-17T15:52:26.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 1 -- 172.21.15.64:0/181412683 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4228893546 0 0) 0x7fb1c8003520 con 0x7fb1e00a54b0 2024-09-17T15:52:26.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 1 -- 172.21.15.64:0/181412683 >> v1:172.21.15.100:6790/0 conn(0x7fb1e01360a0 legacy=0x7fb1e0141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 1 -- 172.21.15.64:0/181412683 >> v1:172.21.15.64:6790/0 conn(0x7fb1e0135c80 legacy=0x7fb1e013df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 1 monclient: found mon.noname-c 2024-09-17T15:52:26.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:52:26.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 1 -- 172.21.15.64:0/181412683 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e009dd40 con 0x7fb1e00a54b0 2024-09-17T15:52:26.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 1 -- 172.21.15.64:0/181412683 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb1d0001830 con 0x7fb1e00a54b0 2024-09-17T15:52:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1e65d0640 5 monclient: authenticate success, global_id 154291 2024-09-17T15:52:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1e65d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 10 monclient: dump: 2024-09-17T15:52:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.270 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1e65d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 1 -- 172.21.15.64:0/181412683 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1d00019d0 con 0x7fb1e00a54b0 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1e65d0640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1e65d0640 4 set_mon_vals no callback set 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 1 -- 172.21.15.64:0/181412683 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb1d0004be0 con 0x7fb1e00a54b0 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1deffd640 10 monclient: dump: 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.272 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1e65d0640 10 monclient: shutdown 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1e65d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1e65d0640 1 -- 172.21.15.64:0/181412683 >> v1:172.21.15.64:6791/0 conn(0x7fb1e00a54b0 legacy=0x7fb1e00a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1e65d0640 1 -- 172.21.15.64:0/181412683 shutdown_connections 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1e65d0640 1 -- 172.21.15.64:0/181412683 >> 172.21.15.64:0/181412683 conn(0x7fb1e006daa0 msgr2=0x7fb1e00ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1e65d0640 1 -- 172.21.15.64:0/181412683 shutdown_connections 2024-09-17T15:52:26.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.258+0000 7fb1e65d0640 1 -- 172.21.15.64:0/181412683 wait complete. 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.259+0000 7fb1e65d0640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.259+0000 7fb1e65d0640 10 monclient: monmap: 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:52:26.274 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.259+0000 7fb1e65d0640 1 Processor -- start 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.259+0000 7fb1e65d0640 1 -- start start 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.259+0000 7fb1e65d0640 10 monclient: init 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1e65d0640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1e65d0640 10 monclient: _add_conns ranks=[5,2,8,6,4,7,1,3,0] 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1e65d0640 10 monclient(hunting): picked mon.noname-h con 0x7fb1e00a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1e65d0640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e0135c80 addr v1:172.21.15.64:6790/0 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1e65d0640 10 monclient(hunting): picked mon.noname-e con 0x7fb1e01360a0 addr v1:172.21.15.64:6793/0 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1e65d0640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e00a1b20 con 0x7fb1e0135c80 2024-09-17T15:52:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1e65d0640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e00a6c80 con 0x7fb1e00a54b0 2024-09-17T15:52:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1e65d0640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e00a3730 con 0x7fb1e01360a0 2024-09-17T15:52:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1e65d0640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1e65d0640 10 monclient(hunting): authenticate will time out at 1143.935404s 2024-09-17T15:52:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1e4b6d640 1 -- >> v1:172.21.15.64:6793/0 conn(0x7fb1e01360a0 legacy=0x7fb1e01d5e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:52:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dffff640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7fb1e00a54b0 legacy=0x7fb1e009fb00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:52:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1df7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fb1e0135c80 0x7fb1e00a0210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:41258/0 (socket says 172.21.15.64:41258) 2024-09-17T15:52:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1df7fe640 1 -- 172.21.15.64:0/1343824879 learned_addr learned my addr 172.21.15.64:0/1343824879 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:52:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 205417695 0 0) 0x7fb1e00a1b20 con 0x7fb1e0135c80 2024-09-17T15:52:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dcff9640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dcff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dcff9640 10 monclient(hunting): my global_id is 124976 2024-09-17T15:52:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1cc003520 con 0x7fb1e0135c80 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1458871280 0 0) 0x7fb1cc003520 con 0x7fb1e0135c80 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1e00a1b20 con 0x7fb1e0135c80 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb1d4001bb0 con 0x7fb1e0135c80 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dcff9640 10 monclient(hunting): dump: 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.277 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.260+0000 7fb1dcff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 92075332 0 0) 0x7fb1e00a1b20 con 0x7fb1e0135c80 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 >> v1:172.21.15.64:6793/0 conn(0x7fb1e01360a0 legacy=0x7fb1e01d5e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 >> v1:172.21.15.100:6791/0 conn(0x7fb1e00a54b0 legacy=0x7fb1e009fb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 1 monclient: found mon.b 2024-09-17T15:52:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e00a0920 con 0x7fb1e0135c80 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1e65d0640 5 monclient: authenticate success, global_id 124976 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1e65d0640 10 monclient: _renew_subs 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1e65d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1e65d0640 1 -- 172.21.15.64:0/1343824879 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb1e00a2e90 con 0x7fb1e0135c80 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1e65d0640 10 monclient: _renew_subs 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1e65d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1e65d0640 1 -- 172.21.15.64:0/1343824879 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb1e00a3c30 con 0x7fb1e0135c80 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1d40038a0 con 0x7fb1e0135c80 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb1d4003e10 con 0x7fb1e0135c80 2024-09-17T15:52:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1de7fc640 4 set_mon_vals no callback set 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1dcff9640 10 monclient: dump: 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.281 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.281 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.281 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.281 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.281 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1e65d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.261+0000 7fb1e65d0640 1 -- 172.21.15.64:0/1343824879 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb1c00065c0 con 0x7fb1c0002860 2024-09-17T15:52:26.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fb1d4003fb0 con 0x7fb1e0135c80 2024-09-17T15:52:26.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(290..290 src has 192..290) ==== 3898+0+0 (unknown 2326613063 0 0) 0x7fb1d404ed40 con 0x7fb1e0135c80 2024-09-17T15:52:26.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1902774719 0 0) 0x7fb1c00065c0 con 0x7fb1c0002860 2024-09-17T15:52:26.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 10 monclient(hunting): my global_id is 124884 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1e00a1b20 con 0x7fb1c0002860 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1704658069 0 0) 0x7fb1e00a1b20 con 0x7fb1c0002860 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1c00065c0 con 0x7fb1c0002860 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1e4b6d640 10 monclient: get_auth_request con 0x7fb1cc040070 auth_method 0 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb1d0004be0 con 0x7fb1c0002860 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 10 monclient: dump: 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.285 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1907426592 0 0) 0x7fb1c00065c0 con 0x7fb1c0002860 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.262+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 --> v1:172.21.15.64:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1c000a160 con 0x7fb1c0002860 2024-09-17T15:52:26.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.263+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb1d00011d0 con 0x7fb1c0002860 2024-09-17T15:52:26.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.263+0000 7fb1dcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.263+0000 7fb1dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:26.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.263+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 >> v1:172.21.15.64:6789/0 conn(0x7fb1c0002860 legacy=0x7fb1c0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.263+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 >> v1:172.21.15.64:6789/0 conn(0x7fb1c0002860 legacy=0x7fb1c0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1e65d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:52:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1e65d0640 1 -- 172.21.15.64:0/1343824879 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb1c0009e70 con 0x7fb1c0009130 2024-09-17T15:52:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1419872849 0 0) 0x7fb1c0009e70 con 0x7fb1c0009130 2024-09-17T15:52:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 10 monclient(hunting): my global_id is 124893 2024-09-17T15:52:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1c00065c0 con 0x7fb1c0009130 2024-09-17T15:52:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2818179566 0 0) 0x7fb1c00065c0 con 0x7fb1c0009130 2024-09-17T15:52:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1c0009e70 con 0x7fb1c0009130 2024-09-17T15:52:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb1d0005000 con 0x7fb1c0009130 2024-09-17T15:52:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 10 monclient: dump: 2024-09-17T15:52:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.289 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2729391794 0 0) 0x7fb1c0009e70 con 0x7fb1c0009130 2024-09-17T15:52:26.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.269+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 --> v1:172.21.15.64:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb1c000c410 con 0x7fb1c0009130 2024-09-17T15:52:26.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7667 (unknown 0 0 3287128631) 0x7fb1d0005130 con 0x7fb1c0009130 2024-09-17T15:52:26.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1dcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:52:26.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:26.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 >> v1:172.21.15.64:6789/0 conn(0x7fb1c0009130 legacy=0x7fb1c00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1dcff9640 1 -- 172.21.15.64:0/1343824879 >> v1:172.21.15.64:6789/0 conn(0x7fb1c0009130 legacy=0x7fb1c00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:26.291 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:52:26.291 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:52:26.291 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:52:26.291 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:52:26.291 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 40, 2024-09-17T15:52:26.292 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 153188, 2024-09-17T15:52:26.292 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:52:26.292 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:52:26.292 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:52:26.292 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:52:26.292 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:52:26.292 INFO:teuthology.orchestra.run.smithi064.stdout: 4 2024-09-17T15:52:26.292 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.292 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 2, 2024-09-17T15:52:26.292 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:52:26.292 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:52:26.292 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:52:26.292 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:52:26.293 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:52:26.294 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:52:26.295 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:52:26.295 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:52:26.295 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:52:26.295 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:52:26.295 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:52:26.295 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:52:26.295 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:52:26.295 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:52:26.295 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:52:26.295 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:52:26.295 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:52:26.295 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:52:26.295 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:52:26.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.297 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:52:26.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.297 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:52:26.297 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:52:26.297 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.297 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.297 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.297 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:52:26.297 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.297 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.298 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:52:26.299 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.299 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.299 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.299 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:52:26.299 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:52:26.299 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.299 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.299 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.299 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:52:26.299 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:52:26.300 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.300 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.300 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.300 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:52:26.300 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.300 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.300 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.300 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:52:26.300 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:52:26.300 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.300 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.302 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:52:26.302 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:52:26.302 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.302 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.302 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.302 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:52:26.302 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:52:26.302 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.302 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.304 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:52:26.304 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:52:26.304 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.304 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.304 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.304 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:52:26.304 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.304 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.304 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:52:26.304 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.306 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.306 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:52:26.306 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:52:26.306 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.306 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.306 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.306 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:52:26.306 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:52:26.306 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.306 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.307 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.307 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:52:26.307 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.307 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.307 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.307 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:52:26.307 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:52:26.307 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.307 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.308 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.308 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:52:26.308 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:52:26.308 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.308 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.308 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.308 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:52:26.308 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.308 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.309 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:52:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:52:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:52:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": [ 2024-09-17T15:52:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.310 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.311 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:52:26.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.311 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.311 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:52:26.311 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:52:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1e65d0640 1 -- 172.21.15.64:0/1343824879 >> v1:172.21.15.100:6812/4210963484 conn(0x7fb1cc040070 legacy=0x7fb1cc042530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1e65d0640 10 monclient: shutdown 2024-09-17T15:52:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1e65d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1e65d0640 1 -- 172.21.15.64:0/1343824879 >> v1:172.21.15.64:6790/0 conn(0x7fb1e0135c80 legacy=0x7fb1e00a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1e4b6d640 1 -- 172.21.15.64:0/1343824879 reap_dead start 2024-09-17T15:52:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1e4b6d640 1 -- 172.21.15.64:0/1343824879 reap_dead start 2024-09-17T15:52:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1e65d0640 1 -- 172.21.15.64:0/1343824879 shutdown_connections 2024-09-17T15:52:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1e65d0640 1 -- 172.21.15.64:0/1343824879 >> 172.21.15.64:0/1343824879 conn(0x7fb1e006daa0 msgr2=0x7fb1e00a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1e65d0640 1 -- 172.21.15.64:0/1343824879 shutdown_connections 2024-09-17T15:52:26.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.270+0000 7fb1e65d0640 1 -- 172.21.15.64:0/1343824879 wait complete. 2024-09-17T15:52:26.313 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:52:26.313 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon b 2024-09-17T15:52:26.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:52:26.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.414+0000 7f082b3b4640 10 monclient: get_monmap_and_config 2024-09-17T15:52:26.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.414+0000 7f082b3b4640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.414+0000 7f082b3b4640 10 monclient: monmap: 2024-09-17T15:52:26.416 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:52:26.416 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.416 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:52:26.415095+0000 2024-09-17T15:52:26.416 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:52:26.415095+0000 2024-09-17T15:52:26.416 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.416 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.417 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:52:26.417 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:52:26.417 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:52:26.417 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:52:26.417 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:52:26.417 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:52:26.417 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:52:26.417 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:52:26.417 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:52:26.417 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.414+0000 7f082b3b4640 1 Processor -- start 2024-09-17T15:52:26.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f082b3b4640 1 -- start start 2024-09-17T15:52:26.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f082b3b4640 10 monclient: init 2024-09-17T15:52:26.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f082b3b4640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f082b3b4640 10 monclient: _add_conns ranks=[4,5,6,8,3,0,1,2,7] 2024-09-17T15:52:26.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f082b3b4640 10 monclient(hunting): picked mon.noname-c con 0x7f08240a54b0 addr v1:172.21.15.64:6791/0 2024-09-17T15:52:26.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f082b3b4640 10 monclient(hunting): picked mon.noname-h con 0x7f0824135c80 addr v1:172.21.15.100:6791/0 2024-09-17T15:52:26.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f082b3b4640 10 monclient(hunting): picked mon.noname-d con 0x7f08241360a0 addr v1:172.21.15.64:6792/0 2024-09-17T15:52:26.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f082b3b4640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08240aacb0 con 0x7f08240a54b0 2024-09-17T15:52:26.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f082b3b4640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f082409d780 con 0x7f0824135c80 2024-09-17T15:52:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f082b3b4640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f082409da60 con 0x7f08241360a0 2024-09-17T15:52:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f082b3b4640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f082b3b4640 10 monclient(hunting): authenticate will time out at 1144.090590s 2024-09-17T15:52:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f0829951640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7f08241360a0 legacy=0x7f0824141640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:52:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f0829150640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f08240a54b0 0x7f08240a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:39840/0 (socket says 172.21.15.64:39840) 2024-09-17T15:52:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f0829150640 1 -- 172.21.15.64:0/220416613 learned_addr learned my addr 172.21.15.64:0/220416613 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:52:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f082894f640 1 -- 172.21.15.64:0/220416613 >> v1:172.21.15.100:6791/0 conn(0x7f0824135c80 legacy=0x7f082413df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:52:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f081bfff640 1 -- 172.21.15.64:0/220416613 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3750766520 0 0) 0x7f08240aacb0 con 0x7f08240a54b0 2024-09-17T15:52:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f081bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f081bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f081bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f081bfff640 10 monclient(hunting): my global_id is 154300 2024-09-17T15:52:26.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.415+0000 7f081bfff640 1 -- 172.21.15.64:0/220416613 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0810003520 con 0x7f08240a54b0 2024-09-17T15:52:26.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 1 -- 172.21.15.64:0/220416613 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3106446514 0 0) 0x7f0810003520 con 0x7f08240a54b0 2024-09-17T15:52:26.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 1 -- 172.21.15.64:0/220416613 >> v1:172.21.15.64:6792/0 conn(0x7f08241360a0 legacy=0x7f0824141640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:26.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 1 -- 172.21.15.64:0/220416613 >> v1:172.21.15.100:6791/0 conn(0x7f0824135c80 legacy=0x7f082413df10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:26.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 1 monclient: found mon.noname-c 2024-09-17T15:52:26.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:52:26.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 1 -- 172.21.15.64:0/220416613 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f082409dd40 con 0x7f08240a54b0 2024-09-17T15:52:26.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 1 -- 172.21.15.64:0/220416613 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0820001830 con 0x7f08240a54b0 2024-09-17T15:52:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 10 monclient: dump: 2024-09-17T15:52:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f082b3b4640 5 monclient: authenticate success, global_id 154300 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f082b3b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 1 -- 172.21.15.64:0/220416613 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0820001a50 con 0x7f08240a54b0 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 1 -- 172.21.15.64:0/220416613 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0820004be0 con 0x7f08240a54b0 2024-09-17T15:52:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f082b3b4640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f082b3b4640 4 set_mon_vals no callback set 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f081bfff640 10 monclient: dump: 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f082b3b4640 10 monclient: shutdown 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f082b3b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f082b3b4640 1 -- 172.21.15.64:0/220416613 >> v1:172.21.15.64:6791/0 conn(0x7f08240a54b0 legacy=0x7f08240a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f082b3b4640 1 -- 172.21.15.64:0/220416613 shutdown_connections 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f082b3b4640 1 -- 172.21.15.64:0/220416613 >> 172.21.15.64:0/220416613 conn(0x7f082406daa0 msgr2=0x7f08240ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f082b3b4640 1 -- 172.21.15.64:0/220416613 shutdown_connections 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.416+0000 7f082b3b4640 1 -- 172.21.15.64:0/220416613 wait complete. 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.417+0000 7f082b3b4640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.417+0000 7f082b3b4640 10 monclient: monmap: 2024-09-17T15:52:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:52:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:52:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:52:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:52:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:52:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:52:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:52:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:52:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:52:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:52:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:52:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:52:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:52:26.433 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.417+0000 7f082b3b4640 1 Processor -- start 2024-09-17T15:52:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.417+0000 7f082b3b4640 1 -- start start 2024-09-17T15:52:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.417+0000 7f082b3b4640 10 monclient: init 2024-09-17T15:52:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f082b3b4640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f082b3b4640 10 monclient: _add_conns ranks=[4,6,3,5,1,8,7,2,0] 2024-09-17T15:52:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f082b3b4640 10 monclient(hunting): picked mon.noname-c con 0x7f08240a54b0 addr v1:172.21.15.64:6791/0 2024-09-17T15:52:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f082b3b4640 10 monclient(hunting): picked mon.noname-d con 0x7f0824135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:52:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f082b3b4640 10 monclient(hunting): picked mon.noname-g con 0x7f08241360a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:52:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f082b3b4640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08240a0170 con 0x7f08241360a0 2024-09-17T15:52:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f082894f640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7f0824135c80 legacy=0x7f082409fa60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f082b3b4640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08240a5120 con 0x7f08240a54b0 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f082b3b4640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08240a1d40 con 0x7f0824135c80 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f082b3b4640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f082b3b4640 10 monclient(hunting): authenticate will time out at 1144.093462s 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0829150640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f08240a54b0 0x7f082409f350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:39844/0 (socket says 172.21.15.64:39844) 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0829150640 1 -- 172.21.15.64:0/129136281 learned_addr learned my addr 172.21.15.64:0/129136281 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1702450769 0 0) 0x7f08240a5120 con 0x7f08240a54b0 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 10 monclient(hunting): my global_id is 154309 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0808003520 con 0x7f08240a54b0 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4097050888 0 0) 0x7f0808003520 con 0x7f08240a54b0 2024-09-17T15:52:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08240a5120 con 0x7f08240a54b0 2024-09-17T15:52:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0820004b20 con 0x7f08240a54b0 2024-09-17T15:52:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:52:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 10 monclient(hunting): dump: 2024-09-17T15:52:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.436 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.436 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.436 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.436 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.436 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.436 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.436 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:52:26.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3684847463 0 0) 0x7f08240a0170 con 0x7f08241360a0 2024-09-17T15:52:26.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 10 monclient(hunting): my global_id is 115644 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.418+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0808003520 con 0x7f08241360a0 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2135244667 0 0) 0x7f08240a5120 con 0x7f08240a54b0 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 >> v1:172.21.15.64:6792/0 conn(0x7f0824135c80 legacy=0x7f082409fa60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 >> v1:172.21.15.100:6790/0 conn(0x7f08241360a0 legacy=0x7f08241d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 1 monclient: found mon.c 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08240a2020 con 0x7f08240a54b0 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f082b3b4640 5 monclient: authenticate success, global_id 154309 2024-09-17T15:52:26.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f082b3b4640 10 monclient: _renew_subs 2024-09-17T15:52:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f082b3b4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f082b3b4640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f08240a1000 con 0x7f08240a54b0 2024-09-17T15:52:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0820004ea0 con 0x7f08240a54b0 2024-09-17T15:52:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0820005390 con 0x7f08240a54b0 2024-09-17T15:52:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 10 monclient: dump: 2024-09-17T15:52:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.438 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.439 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f081b7fe640 4 set_mon_vals no callback set 2024-09-17T15:52:26.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f082b3b4640 10 monclient: _renew_subs 2024-09-17T15:52:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f082b3b4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f082b3b4640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f0824132920 con 0x7f08240a54b0 2024-09-17T15:52:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.419+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f0820011c70 con 0x7f08240a54b0 2024-09-17T15:52:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f082b3b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f082b3b4640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f08240ab550 con 0x7f082412f460 2024-09-17T15:52:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(290..290 src has 192..290) ==== 3898+0+0 (unknown 2326613063 0 0) 0x7f082004ed20 con 0x7f08240a54b0 2024-09-17T15:52:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1918741103 0 0) 0x7f08240ab550 con 0x7f082412f460 2024-09-17T15:52:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 10 monclient(hunting): my global_id is 124985 2024-09-17T15:52:26.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08240a5120 con 0x7f082412f460 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0829951640 10 monclient: get_auth_request con 0x7f0808040170 auth_method 0 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2444231931 0 0) 0x7f08240a5120 con 0x7f082412f460 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08240ab550 con 0x7f082412f460 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f08140041c0 con 0x7f082412f460 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 10 monclient: dump: 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.441 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4293290506 0 0) 0x7f08240ab550 con 0x7f082412f460 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.420+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.64:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f08240a67b0 con 0x7f082412f460 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.421+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0814006480 con 0x7f082412f460 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.421+0000 7f0819ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.421+0000 7f0819ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:26.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.421+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 >> v1:172.21.15.64:6790/0 conn(0x7f082412f460 legacy=0x7f082412f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.421+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 >> v1:172.21.15.64:6790/0 conn(0x7f082412f460 legacy=0x7f082412f840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f082b3b4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:52:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f082b3b4640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0824132cb0 con 0x7f082412ff50 2024-09-17T15:52:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2862760264 0 0) 0x7f0824132cb0 con 0x7f082412ff50 2024-09-17T15:52:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f0819ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f0819ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f0819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f0819ffb640 10 monclient(hunting): my global_id is 124994 2024-09-17T15:52:26.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08240ab550 con 0x7f082412ff50 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4188477292 0 0) 0x7f08240ab550 con 0x7f082412ff50 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0824132cb0 con 0x7f082412ff50 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0814005cf0 con 0x7f082412ff50 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f0819ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f0819ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.427+0000 7f0819ffb640 10 monclient: dump: 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1904490616 0 0) 0x7f0824132cb0 con 0x7f082412ff50 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 --> v1:172.21.15.64:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0824068b30 con 0x7f082412ff50 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7667 (unknown 0 0 4021089100) 0x7f0814005e20 con 0x7f082412ff50 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f0819ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f0819ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 >> v1:172.21.15.64:6790/0 conn(0x7f082412ff50 legacy=0x7f08241db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f0819ffb640 1 -- 172.21.15.64:0/129136281 >> v1:172.21.15.64:6790/0 conn(0x7f082412ff50 legacy=0x7f08241db560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:26.447 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:52:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:52:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:52:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:52:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 40, 2024-09-17T15:52:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 153346, 2024-09-17T15:52:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:52:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:52:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:52:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:52:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:52:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: 4 2024-09-17T15:52:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 2, 2024-09-17T15:52:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:52:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:52:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:52:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:52:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:52:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:52:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:52:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:52:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:52:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:52:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:52:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:52:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:52:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:52:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:52:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:52:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:52:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:52:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:52:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:52:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:52:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:52:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:52:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:52:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:52:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:52:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:52:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:52:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:52:26.466 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:52:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f082b3b4640 1 -- 172.21.15.64:0/129136281 >> v1:172.21.15.100:6812/4210963484 conn(0x7f0808040170 legacy=0x7f0808042630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f082b3b4640 10 monclient: shutdown 2024-09-17T15:52:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f082b3b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f082b3b4640 1 -- 172.21.15.64:0/129136281 >> v1:172.21.15.64:6791/0 conn(0x7f08240a54b0 legacy=0x7f082409f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f0829951640 1 -- 172.21.15.64:0/129136281 reap_dead start 2024-09-17T15:52:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f0829951640 1 -- 172.21.15.64:0/129136281 reap_dead start 2024-09-17T15:52:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f082b3b4640 1 -- 172.21.15.64:0/129136281 shutdown_connections 2024-09-17T15:52:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f082b3b4640 1 -- 172.21.15.64:0/129136281 >> 172.21.15.64:0/129136281 conn(0x7f082406daa0 msgr2=0x7f08240a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f082b3b4640 1 -- 172.21.15.64:0/129136281 shutdown_connections 2024-09-17T15:52:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.428+0000 7f082b3b4640 1 -- 172.21.15.64:0/129136281 wait complete. 2024-09-17T15:52:26.468 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:52:26.468 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon c 2024-09-17T15:52:26.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:52:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.572+0000 7f3927b5d640 10 monclient: get_monmap_and_config 2024-09-17T15:52:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.572+0000 7f3927b5d640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.572+0000 7f3927b5d640 10 monclient: monmap: 2024-09-17T15:52:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:52:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:52:26.573198+0000 2024-09-17T15:52:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:52:26.573198+0000 2024-09-17T15:52:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.574 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.575 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:52:26.575 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:52:26.575 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:52:26.575 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:52:26.575 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:52:26.575 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:52:26.575 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:52:26.575 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:52:26.575 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:52:26.575 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.572+0000 7f3927b5d640 1 Processor -- start 2024-09-17T15:52:26.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f3927b5d640 1 -- start start 2024-09-17T15:52:26.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f3927b5d640 10 monclient: init 2024-09-17T15:52:26.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f3927b5d640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f3927b5d640 10 monclient: _add_conns ranks=[4,2,8,5,3,7,0,1,6] 2024-09-17T15:52:26.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f3927b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f39200a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:52:26.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f3927b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f3920135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:52:26.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f3927b5d640 10 monclient(hunting): picked mon.noname-e con 0x7f3920135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:52:26.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f3927b5d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f392009d6f0 con 0x7f3920135b50 2024-09-17T15:52:26.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f3927b5d640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f392009d9d0 con 0x7f39200a5520 2024-09-17T15:52:26.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f3927b5d640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f392009dcb0 con 0x7f3920135f70 2024-09-17T15:52:26.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f3927b5d640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f3927b5d640 10 monclient(hunting): authenticate will time out at 1144.248725s 2024-09-17T15:52:26.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f39260fa640 1 -- >> v1:172.21.15.64:6793/0 conn(0x7f3920135f70 legacy=0x7f39201415e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:52:26.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f39258f9640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f39200a5520 0x7f39200a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:39852/0 (socket says 172.21.15.64:39852) 2024-09-17T15:52:26.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f39258f9640 1 -- 172.21.15.64:0/1219630532 learned_addr learned my addr 172.21.15.64:0/1219630532 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:52:26.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.573+0000 7f39250f8640 1 --1- 172.21.15.64:0/1219630532 >> v1:172.21.15.64:6790/0 conn(0x7f3920135b50 0x7f392013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:41278/0 (socket says 172.21.15.64:41278) 2024-09-17T15:52:26.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 1 -- 172.21.15.64:0/1219630532 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 592710168 0 0) 0x7f392009d6f0 con 0x7f3920135b50 2024-09-17T15:52:26.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient(hunting): my global_id is 125003 2024-09-17T15:52:26.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 1 -- 172.21.15.64:0/1219630532 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f390c003520 con 0x7f3920135b50 2024-09-17T15:52:26.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 1 -- 172.21.15.64:0/1219630532 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2114171035 0 0) 0x7f392009d9d0 con 0x7f39200a5520 2024-09-17T15:52:26.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient(hunting): my global_id is 154318 2024-09-17T15:52:26.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 1 -- 172.21.15.64:0/1219630532 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f392009d6f0 con 0x7f39200a5520 2024-09-17T15:52:26.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 1 -- 172.21.15.64:0/1219630532 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 191398081 0 0) 0x7f392009d6f0 con 0x7f39200a5520 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 1 -- 172.21.15.64:0/1219630532 >> v1:172.21.15.64:6793/0 conn(0x7f3920135f70 legacy=0x7f39201415e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 1 -- 172.21.15.64:0/1219630532 >> v1:172.21.15.64:6790/0 conn(0x7f3920135b50 legacy=0x7f392013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 1 monclient: found mon.noname-c 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 1 -- 172.21.15.64:0/1219630532 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f392009ccd0 con 0x7f39200a5520 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 1 -- 172.21.15.64:0/1219630532 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3914001810 con 0x7f39200a5520 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f3927b5d640 5 monclient: authenticate success, global_id 154318 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f3927b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:26.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:26.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient: dump: 2024-09-17T15:52:26.587 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.587 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.587 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.587 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.587 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.587 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.587 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.587 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.587 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.587 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.587 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f3927b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 1 -- 172.21.15.64:0/1219630532 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f39140019b0 con 0x7f39200a5520 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f3927b5d640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 1 -- 172.21.15.64:0/1219630532 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3914004be0 con 0x7f39200a5520 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f3927b5d640 4 set_mon_vals no callback set 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:26.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f39248f7640 10 monclient: dump: 2024-09-17T15:52:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f3927b5d640 10 monclient: shutdown 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f3927b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.574+0000 7f3927b5d640 1 -- 172.21.15.64:0/1219630532 >> v1:172.21.15.64:6791/0 conn(0x7f39200a5520 legacy=0x7f39200a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.575+0000 7f3927b5d640 1 -- 172.21.15.64:0/1219630532 shutdown_connections 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.575+0000 7f3927b5d640 1 -- 172.21.15.64:0/1219630532 >> 172.21.15.64:0/1219630532 conn(0x7f392006db30 msgr2=0x7f39200ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.575+0000 7f3927b5d640 1 -- 172.21.15.64:0/1219630532 shutdown_connections 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.575+0000 7f3927b5d640 1 -- 172.21.15.64:0/1219630532 wait complete. 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.575+0000 7f3927b5d640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.575+0000 7f3927b5d640 10 monclient: monmap: 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:52:26.590 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:26.591 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:52:26.591 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:52:26.591 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.591 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:52:26.591 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:52:26.591 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:52:26.591 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:52:26.591 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:52:26.591 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:52:26.591 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:52:26.591 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:52:26.591 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:52:26.592 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:52:26.592 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.575+0000 7f3927b5d640 1 Processor -- start 2024-09-17T15:52:26.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f3927b5d640 1 -- start start 2024-09-17T15:52:26.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f3927b5d640 10 monclient: init 2024-09-17T15:52:26.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f3927b5d640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f3927b5d640 10 monclient: _add_conns ranks=[4,6,0,2,1,8,3,7,5] 2024-09-17T15:52:26.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f3927b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f39200a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:52:26.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f3927b5d640 10 monclient(hunting): picked mon.noname-d con 0x7f3920135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:52:26.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f3927b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f3920135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:52:26.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f3927b5d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f392009e680 con 0x7f3920135f70 2024-09-17T15:52:26.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f3927b5d640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39200a0e00 con 0x7f39200a5520 2024-09-17T15:52:26.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39250f8640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7f3920135b50 legacy=0x7f3920132260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:52:26.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f3927b5d640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39200a9280 con 0x7f3920135b50 2024-09-17T15:52:26.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f3927b5d640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f3927b5d640 10 monclient(hunting): authenticate will time out at 1144.251532s 2024-09-17T15:52:26.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39258f9640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f39200a5520 0x7f3920131b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:39860/0 (socket says 172.21.15.64:39860) 2024-09-17T15:52:26.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39258f9640 1 -- 172.21.15.64:0/575290680 learned_addr learned my addr 172.21.15.64:0/575290680 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:52:26.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588393266 0 0) 0x7f392009ccd0 con 0x7f39200a5520 2024-09-17T15:52:26.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39127fc640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39127fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39127fc640 10 monclient(hunting): my global_id is 154327 2024-09-17T15:52:26.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3904003520 con 0x7f39200a5520 2024-09-17T15:52:26.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1599751154 0 0) 0x7f392009e680 con 0x7f3920135f70 2024-09-17T15:52:26.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39127fc640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39127fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39127fc640 10 monclient(hunting): my global_id is 124902 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f392009ccd0 con 0x7f3920135f70 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.576+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1547338793 0 0) 0x7f3904003520 con 0x7f39200a5520 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f392009e680 con 0x7f39200a5520 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3914004b20 con 0x7f39200a5520 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient(hunting): dump: 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.594 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 914014768 0 0) 0x7f392009ccd0 con 0x7f3920135f70 2024-09-17T15:52:26.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3904003520 con 0x7f3920135f70 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f3918004550 con 0x7f3920135f70 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient(hunting): dump: 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.596 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.597 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.597 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.597 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.597 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.597 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.597 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 410873807 0 0) 0x7f392009e680 con 0x7f39200a5520 2024-09-17T15:52:26.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 >> v1:172.21.15.64:6792/0 conn(0x7f3920135b50 legacy=0x7f3920132260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:52:26.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 >> v1:172.21.15.64:6789/0 conn(0x7f3920135f70 legacy=0x7f39201d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 1 monclient: found mon.c 2024-09-17T15:52:26.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3920132970 con 0x7f39200a5520 2024-09-17T15:52:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f3927b5d640 5 monclient: authenticate success, global_id 154327 2024-09-17T15:52:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f3927b5d640 10 monclient: _renew_subs 2024-09-17T15:52:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f3927b5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f3927b5d640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f39200a09e0 con 0x7f39200a5520 2024-09-17T15:52:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f3927b5d640 10 monclient: _renew_subs 2024-09-17T15:52:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f3927b5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f3927b5d640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f39200a51e0 con 0x7f39200a5520 2024-09-17T15:52:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f39140018f0 con 0x7f39200a5520 2024-09-17T15:52:26.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f39140051e0 con 0x7f39200a5520 2024-09-17T15:52:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f3913fff640 4 set_mon_vals no callback set 2024-09-17T15:52:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.577+0000 7f39127fc640 10 monclient: dump: 2024-09-17T15:52:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.599 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f3914011ac0 con 0x7f39200a5520 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f3927b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f3927b5d640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f39080065c0 con 0x7f3908002860 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(291..291 src has 192..291) ==== 3898+0+0 (unknown 1672347257 0 0) 0x7f391404ed30 con 0x7f39200a5520 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1001849818 0 0) 0x7f39080065c0 con 0x7f3908002860 2024-09-17T15:52:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 10 monclient(hunting): my global_id is 154336 2024-09-17T15:52:26.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f392009e680 con 0x7f3908002860 2024-09-17T15:52:26.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3545145099 0 0) 0x7f392009e680 con 0x7f3908002860 2024-09-17T15:52:26.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f39080065c0 con 0x7f3908002860 2024-09-17T15:52:26.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39260fa640 10 monclient: get_auth_request con 0x7f39040408e0 auth_method 0 2024-09-17T15:52:26.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f391c0016a0 con 0x7f3908002860 2024-09-17T15:52:26.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:26.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 10 monclient: dump: 2024-09-17T15:52:26.601 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.601 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.602 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.602 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.602 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.602 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.602 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.602 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.602 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.602 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.602 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.602 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.602 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.602 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.602 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.603 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 256793909 0 0) 0x7f39080065c0 con 0x7f3908002860 2024-09-17T15:52:26.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.578+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f390800a160 con 0x7f3908002860 2024-09-17T15:52:26.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.579+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f391c001c50 con 0x7f3908002860 2024-09-17T15:52:26.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.579+0000 7f39127fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.579+0000 7f39127fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:26.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.579+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 >> v1:172.21.15.64:6791/0 conn(0x7f3908002860 legacy=0x7f3908004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.579+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 >> v1:172.21.15.64:6791/0 conn(0x7f3908002860 legacy=0x7f3908004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:26.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f3927b5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:52:26.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f3927b5d640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3908009e50 con 0x7f3908009110 2024-09-17T15:52:26.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3261235800 0 0) 0x7f3908009e50 con 0x7f3908009110 2024-09-17T15:52:26.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f39127fc640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f39127fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f39127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f39127fc640 10 monclient(hunting): my global_id is 154345 2024-09-17T15:52:26.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39080065c0 con 0x7f3908009110 2024-09-17T15:52:26.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3769416251 0 0) 0x7f39080065c0 con 0x7f3908009110 2024-09-17T15:52:26.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3908009e50 con 0x7f3908009110 2024-09-17T15:52:26.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f391c004d10 con 0x7f3908009110 2024-09-17T15:52:26.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f39127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f39127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.585+0000 7f39127fc640 10 monclient: dump: 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.606 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.607 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.607 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.607 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3589128113 0 0) 0x7f3908009e50 con 0x7f3908009110 2024-09-17T15:52:26.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 --> v1:172.21.15.64:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f390800c3f0 con 0x7f3908009110 2024-09-17T15:52:26.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7666 (unknown 0 0 3959396962) 0x7f391c006e00 con 0x7f3908009110 2024-09-17T15:52:26.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f39127fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:52:26.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f39127fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:26.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 >> v1:172.21.15.64:6791/0 conn(0x7f3908009110 legacy=0x7f39080095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f39127fc640 1 -- 172.21.15.64:0/575290680 >> v1:172.21.15.64:6791/0 conn(0x7f3908009110 legacy=0x7f39080095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:26.607 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:52:26.608 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:52:26.608 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:52:26.608 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:52:26.608 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 40, 2024-09-17T15:52:26.608 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 37920, 2024-09-17T15:52:26.608 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:52:26.608 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:52:26.608 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:52:26.608 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:52:26.608 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:52:26.608 INFO:teuthology.orchestra.run.smithi064.stdout: 4 2024-09-17T15:52:26.608 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.608 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 3, 2024-09-17T15:52:26.609 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:52:26.609 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:52:26.609 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:52:26.609 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.609 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.609 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.609 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.609 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.609 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.609 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.609 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.609 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.610 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.610 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.610 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.610 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:52:26.610 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:52:26.610 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.610 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.610 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.610 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.610 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.610 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.610 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.610 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:52:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:52:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:52:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:52:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:52:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:52:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:52:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:52:26.612 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:52:26.612 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:52:26.612 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:52:26.612 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:52:26.612 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:52:26.612 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:52:26.612 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:52:26.612 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:52:26.612 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:52:26.612 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.612 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.612 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:52:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:52:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:52:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:52:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:52:26.614 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.615 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.615 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.615 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:52:26.615 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:52:26.615 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.615 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.615 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.615 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:52:26.615 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.616 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.616 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:52:26.616 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:52:26.616 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.616 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.616 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.616 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:52:26.616 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:52:26.616 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:52:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:52:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:52:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:52:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.619 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.620 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.621 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:52:26.621 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.621 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.621 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:52:26.621 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:52:26.621 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.621 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.621 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.621 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:52:26.622 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:52:26.622 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.622 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.622 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.622 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:52:26.622 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.622 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.622 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:52:26.622 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:52:26.622 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:52:26.624 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:52:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.626 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.626 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.626 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:52:26.626 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.626 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.626 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:52:26.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.626 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.626 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.626 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:52:26.626 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.626 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:52:26.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.627 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.627 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.627 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 6 2024-09-17T15:52:26.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.627 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.627 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:52:26.627 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:52:26.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f3927b5d640 1 -- 172.21.15.64:0/575290680 >> v1:172.21.15.100:6812/4210963484 conn(0x7f39040408e0 legacy=0x7f3904042da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f3927b5d640 10 monclient: shutdown 2024-09-17T15:52:26.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f3927b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f3927b5d640 1 -- 172.21.15.64:0/575290680 >> v1:172.21.15.64:6791/0 conn(0x7f39200a5520 legacy=0x7f3920131b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f39260fa640 1 -- 172.21.15.64:0/575290680 reap_dead start 2024-09-17T15:52:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f39260fa640 1 -- 172.21.15.64:0/575290680 reap_dead start 2024-09-17T15:52:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f3927b5d640 1 -- 172.21.15.64:0/575290680 shutdown_connections 2024-09-17T15:52:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f3927b5d640 1 -- 172.21.15.64:0/575290680 >> 172.21.15.64:0/575290680 conn(0x7f392006db30 msgr2=0x7f392012c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f3927b5d640 1 -- 172.21.15.64:0/575290680 shutdown_connections 2024-09-17T15:52:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.586+0000 7f3927b5d640 1 -- 172.21.15.64:0/575290680 wait complete. 2024-09-17T15:52:26.629 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:52:26.629 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon f 2024-09-17T15:52:26.630 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-17T15:52:26.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.729+0000 7f7fbc840640 10 monclient: get_monmap_and_config 2024-09-17T15:52:26.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.729+0000 7f7fbc840640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.730+0000 7f7fbc840640 10 monclient: monmap: 2024-09-17T15:52:26.732 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:52:26.732 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.732 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:52:26.730841+0000 2024-09-17T15:52:26.732 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:52:26.730841+0000 2024-09-17T15:52:26.732 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.732 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.733 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:52:26.733 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:52:26.733 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:52:26.733 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:52:26.733 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:52:26.733 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:52:26.733 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:52:26.733 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:52:26.733 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:52:26.733 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.730+0000 7f7fbc840640 1 Processor -- start 2024-09-17T15:52:26.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.730+0000 7f7fbc840640 1 -- start start 2024-09-17T15:52:26.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.730+0000 7f7fbc840640 10 monclient: init 2024-09-17T15:52:26.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 10 monclient: _add_conns ranks=[0,4,7,6,5,1,3,2,8] 2024-09-17T15:52:26.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 10 monclient(hunting): picked mon.noname-a con 0x7f7fb80a54b0 addr v1:172.21.15.64:6789/0 2024-09-17T15:52:26.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 10 monclient(hunting): picked mon.noname-c con 0x7f7fb8135c80 addr v1:172.21.15.64:6791/0 2024-09-17T15:52:26.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 10 monclient(hunting): picked mon.noname-i con 0x7f7fb81360a0 addr v1:172.21.15.100:6792/0 2024-09-17T15:52:26.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb80aacb0 con 0x7f7fb80a54b0 2024-09-17T15:52:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb809d780 con 0x7f7fb8135c80 2024-09-17T15:52:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb809da60 con 0x7f7fb81360a0 2024-09-17T15:52:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 10 monclient(hunting): authenticate will time out at 1144.406168s 2024-09-17T15:52:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb659c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f7fb80a54b0 0x7f7fb80a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:52616/0 (socket says 172.21.15.64:52616) 2024-09-17T15:52:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb659c640 1 -- 172.21.15.64:0/1293025534 learned_addr learned my addr 172.21.15.64:0/1293025534 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:52:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb6d9d640 1 -- 172.21.15.64:0/1293025534 >> v1:172.21.15.100:6792/0 conn(0x7f7fb81360a0 legacy=0x7f7fb8141640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:52:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 1 -- 172.21.15.64:0/1293025534 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4188980508 0 0) 0x7f7fb80aacb0 con 0x7f7fb80a54b0 2024-09-17T15:52:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient(hunting): my global_id is 124911 2024-09-17T15:52:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 1 -- 172.21.15.64:0/1293025534 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fa4003520 con 0x7f7fb80a54b0 2024-09-17T15:52:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 1 -- 172.21.15.64:0/1293025534 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3650817700 0 0) 0x7f7fb809d780 con 0x7f7fb8135c80 2024-09-17T15:52:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient(hunting): my global_id is 154354 2024-09-17T15:52:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 1 -- 172.21.15.64:0/1293025534 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fb80aacb0 con 0x7f7fb8135c80 2024-09-17T15:52:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 1 -- 172.21.15.64:0/1293025534 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1869482658 0 0) 0x7f7fa4003520 con 0x7f7fb80a54b0 2024-09-17T15:52:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 1 -- 172.21.15.64:0/1293025534 >> v1:172.21.15.100:6792/0 conn(0x7f7fb81360a0 legacy=0x7f7fb8141640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 1 -- 172.21.15.64:0/1293025534 >> v1:172.21.15.64:6791/0 conn(0x7f7fb8135c80 legacy=0x7f7fb813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 1 monclient: found mon.noname-a 2024-09-17T15:52:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:52:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 1 -- 172.21.15.64:0/1293025534 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb809dd40 con 0x7f7fb80a54b0 2024-09-17T15:52:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 5 monclient: authenticate success, global_id 124911 2024-09-17T15:52:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 1 -- 172.21.15.64:0/1293025534 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7fa8001830 con 0x7f7fb80a54b0 2024-09-17T15:52:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient: dump: 2024-09-17T15:52:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.744 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.745 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.745 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:26.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 1 -- 172.21.15.64:0/1293025534 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7fa80019d0 con 0x7f7fb80a54b0 2024-09-17T15:52:26.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 1 -- 172.21.15.64:0/1293025534 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7fa8004be0 con 0x7f7fb80a54b0 2024-09-17T15:52:26.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:26.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fbc840640 4 set_mon_vals no callback set 2024-09-17T15:52:26.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:26.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.731+0000 7f7fb5d9b640 10 monclient: dump: 2024-09-17T15:52:26.746 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.746 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.746 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.746 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.732+0000 7f7fbc840640 10 monclient: shutdown 2024-09-17T15:52:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.732+0000 7f7fbc840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.732+0000 7f7fbc840640 1 -- 172.21.15.64:0/1293025534 >> v1:172.21.15.64:6789/0 conn(0x7f7fb80a54b0 legacy=0x7f7fb80a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.732+0000 7f7fbc840640 1 -- 172.21.15.64:0/1293025534 shutdown_connections 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.732+0000 7f7fbc840640 1 -- 172.21.15.64:0/1293025534 >> 172.21.15.64:0/1293025534 conn(0x7f7fb806daa0 msgr2=0x7f7fb80ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.732+0000 7f7fbc840640 1 -- 172.21.15.64:0/1293025534 shutdown_connections 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.732+0000 7f7fbc840640 1 -- 172.21.15.64:0/1293025534 wait complete. 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.733+0000 7f7fbc840640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.733+0000 7f7fbc840640 10 monclient: monmap: 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:52:26.748 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:52:26.749 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:52:26.749 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:52:26.749 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:52:26.749 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:52:26.749 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:52:26.749 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:52:26.749 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:52:26.749 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:52:26.749 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.733+0000 7f7fbc840640 1 Processor -- start 2024-09-17T15:52:26.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.733+0000 7f7fbc840640 1 -- start start 2024-09-17T15:52:26.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.733+0000 7f7fbc840640 10 monclient: init 2024-09-17T15:52:26.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 10 monclient: _add_conns ranks=[2,3,7,4,5,0,8,6,1] 2024-09-17T15:52:26.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 10 monclient(hunting): picked mon.noname-b con 0x7f7fb80a54b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:52:26.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 10 monclient(hunting): picked mon.noname-g con 0x7f7fb8135c80 addr v1:172.21.15.100:6790/0 2024-09-17T15:52:26.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 10 monclient(hunting): picked mon.noname-i con 0x7f7fb81360a0 addr v1:172.21.15.100:6792/0 2024-09-17T15:52:26.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb809ffb0 con 0x7f7fb80a54b0 2024-09-17T15:52:26.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb80a0510 con 0x7f7fb8135c80 2024-09-17T15:52:26.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb80a07f0 con 0x7f7fb81360a0 2024-09-17T15:52:26.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 10 monclient(hunting): authenticate will time out at 1144.409163s 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fb659c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f7fb80a54b0 0x7f7fb809e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:41284/0 (socket says 172.21.15.64:41284) 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fb659c640 1 -- 172.21.15.64:0/4059592488 learned_addr learned my addr 172.21.15.64:0/4059592488 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fb6d9d640 1 -- 172.21.15.64:0/4059592488 >> v1:172.21.15.100:6792/0 conn(0x7f7fb81360a0 legacy=0x7f7fb81d6030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.100:6792/0 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1594653362 0 0) 0x7f7fb809dd40 con 0x7f7fb80a54b0 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient(hunting): my global_id is 125012 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f98003520 con 0x7f7fb80a54b0 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2493604792 0 0) 0x7f7f98003520 con 0x7f7fb80a54b0 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7fb809dd40 con 0x7f7fb80a54b0 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1187883311 0 0) 0x7f7fb80a0510 con 0x7f7fb8135c80 2024-09-17T15:52:26.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient(hunting): my global_id is 115653 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f98003520 con 0x7f7fb8135c80 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7fa80013b0 con 0x7f7fb80a54b0 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient(hunting): dump: 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.752 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.753 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.753 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.753 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.753 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.753 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.753 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.753 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.753 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.753 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.753 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:52:26.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2038836771 0 0) 0x7f7fb809dd40 con 0x7f7fb80a54b0 2024-09-17T15:52:26.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 >> v1:172.21.15.100:6792/0 conn(0x7f7fb81360a0 legacy=0x7f7fb81d6030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 >> v1:172.21.15.100:6790/0 conn(0x7f7fb8135c80 legacy=0x7f7fb809ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 1 monclient: found mon.b 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb809f120 con 0x7f7fb80a54b0 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7faf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 5 monclient: authenticate success, global_id 125012 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 10 monclient: _renew_subs 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb80a7360 con 0x7f7fb80a54b0 2024-09-17T15:52:26.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 10 monclient: _renew_subs 2024-09-17T15:52:26.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:52:26.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.734+0000 7f7fbc840640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7fb8131ee0 con 0x7f7fb80a54b0 2024-09-17T15:52:26.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.735+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7fa80018a0 con 0x7f7fb80a54b0 2024-09-17T15:52:26.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.735+0000 7f7faf7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.735+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7fa80051e0 con 0x7f7fb80a54b0 2024-09-17T15:52:26.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.735+0000 7f7faf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.735+0000 7f7fb559a640 4 set_mon_vals no callback set 2024-09-17T15:52:26.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.735+0000 7f7faf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:52:26.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.735+0000 7f7faf7fe640 10 monclient: dump: 2024-09-17T15:52:26.755 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.755 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.755 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.755 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.756 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.756 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.756 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.756 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.756 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.756 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.756 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.756 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.756 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.756 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.756 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.756 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.735+0000 7f7fbc840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.735+0000 7f7fbc840640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7f9c0065c0 con 0x7f7f9c002860 2024-09-17T15:52:26.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.735+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f7fa8011ac0 con 0x7f7fb80a54b0 2024-09-17T15:52:26.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(291..291 src has 192..291) ==== 3898+0+0 (unknown 1672347257 0 0) 0x7f7fa804ec50 con 0x7f7fb80a54b0 2024-09-17T15:52:26.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1302944828 0 0) 0x7f7f9c0065c0 con 0x7f7f9c002860 2024-09-17T15:52:26.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 10 monclient(hunting): my global_id is 6706 2024-09-17T15:52:26.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fb809dd40 con 0x7f7f9c002860 2024-09-17T15:52:26.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7fb6d9d640 10 monclient: get_auth_request con 0x7f7f98040240 auth_method 0 2024-09-17T15:52:26.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1065208175 0 0) 0x7f7fb809dd40 con 0x7f7f9c002860 2024-09-17T15:52:26.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f9c0065c0 con 0x7f7f9c002860 2024-09-17T15:52:26.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7fa0001550 con 0x7f7f9c002860 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 10 monclient: dump: 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.759 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.759 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.759 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.759 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 893649438 0 0) 0x7f7f9c0065c0 con 0x7f7f9c002860 2024-09-17T15:52:26.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.736+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.100:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f9c00a160 con 0x7f7f9c002860 2024-09-17T15:52:26.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.737+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7fa0001770 con 0x7f7f9c002860 2024-09-17T15:52:26.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.737+0000 7f7faf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.737+0000 7f7faf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:26.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.737+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 >> v1:172.21.15.100:6789/0 conn(0x7f7f9c002860 legacy=0x7f7f9c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.737+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 >> v1:172.21.15.100:6789/0 conn(0x7f7f9c002860 legacy=0x7f7f9c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:26.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7fbc840640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:52:26.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7fbc840640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7f9c009e50 con 0x7f7f9c009110 2024-09-17T15:52:26.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4289579511 0 0) 0x7f7f9c009e50 con 0x7f7f9c009110 2024-09-17T15:52:26.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7faf7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7faf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7faf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7faf7fe640 10 monclient(hunting): my global_id is 6715 2024-09-17T15:52:26.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f9c0065c0 con 0x7f7f9c009110 2024-09-17T15:52:26.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2387827266 0 0) 0x7f7f9c0065c0 con 0x7f7f9c009110 2024-09-17T15:52:26.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f9c009e50 con 0x7f7f9c009110 2024-09-17T15:52:26.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7fa0003ba0 con 0x7f7f9c009110 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7faf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7faf7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.743+0000 7f7faf7fe640 10 monclient: dump: 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.763 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1978726399 0 0) 0x7f7f9c009e50 con 0x7f7f9c009110 2024-09-17T15:52:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 --> v1:172.21.15.100:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7f9c00c3f0 con 0x7f7f9c009110 2024-09-17T15:52:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7502 (unknown 0 0 2944520278) 0x7f7fa0005090 con 0x7f7f9c009110 2024-09-17T15:52:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7faf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:52:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7faf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 >> v1:172.21.15.100:6789/0 conn(0x7f7f9c009110 legacy=0x7f7f9c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7faf7fe640 1 -- 172.21.15.64:0/4059592488 >> v1:172.21.15.100:6789/0 conn(0x7f7f9c009110 legacy=0x7f7f9c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:26.764 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:52:26.764 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:52:26.764 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:52:26.764 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "leader", 2024-09-17T15:52:26.764 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 40, 2024-09-17T15:52:26.764 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 403394, 2024-09-17T15:52:26.764 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:52:26.764 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:52:26.764 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:52:26.764 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:52:26.764 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:52:26.764 INFO:teuthology.orchestra.run.smithi064.stdout: 4 2024-09-17T15:52:26.764 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 3, 2024-09-17T15:52:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:52:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:52:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:52:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.766 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.766 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.766 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.766 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:52:26.766 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:52:26.766 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.766 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.766 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.766 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.766 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.766 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.766 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:52:26.767 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.768 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.769 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.769 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.769 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.769 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.769 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.769 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.769 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.769 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:52:26.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.769 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:52:26.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:52:26.770 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.771 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.771 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.771 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:52:26.771 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:52:26.771 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.771 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.771 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.771 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:52:26.771 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.772 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.772 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.772 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:52:26.772 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:52:26.772 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.772 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.774 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:52:26.775 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.775 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.775 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.775 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:52:26.775 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:52:26.775 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.775 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.775 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.775 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:52:26.775 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.776 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.777 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:52:26.778 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.781 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:52:26.781 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:52:26.781 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.781 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.781 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.781 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:52:26.781 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.781 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.781 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:52:26.782 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:52:26.782 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.782 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.782 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.782 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.782 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.782 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:52:26.782 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:52:26.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.782 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.782 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.782 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:52:26.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.783 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.783 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:52:26.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.783 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.783 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.783 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:52:26.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.783 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.783 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:52:26.783 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:52:26.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7fbc840640 1 -- 172.21.15.64:0/4059592488 >> v1:172.21.15.100:6812/4210963484 conn(0x7f7f98040240 legacy=0x7f7f98042700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7fbc840640 10 monclient: shutdown 2024-09-17T15:52:26.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7fbc840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7fbc840640 1 -- 172.21.15.64:0/4059592488 >> v1:172.21.15.64:6790/0 conn(0x7f7fb80a54b0 legacy=0x7f7fb809e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7fb6d9d640 1 -- 172.21.15.64:0/4059592488 reap_dead start 2024-09-17T15:52:26.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7fb6d9d640 1 -- 172.21.15.64:0/4059592488 reap_dead start 2024-09-17T15:52:26.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7fbc840640 1 -- 172.21.15.64:0/4059592488 shutdown_connections 2024-09-17T15:52:26.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7fbc840640 1 -- 172.21.15.64:0/4059592488 >> 172.21.15.64:0/4059592488 conn(0x7f7fb806daa0 msgr2=0x7f7fb80a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7fbc840640 1 -- 172.21.15.64:0/4059592488 shutdown_connections 2024-09-17T15:52:26.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.744+0000 7f7fbc840640 1 -- 172.21.15.64:0/4059592488 wait complete. 2024-09-17T15:52:26.786 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:52:26.786 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon g 2024-09-17T15:52:26.786 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-17T15:52:26.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.886+0000 7f6046241640 10 monclient: get_monmap_and_config 2024-09-17T15:52:26.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.886+0000 7f6046241640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.886+0000 7f6046241640 10 monclient: monmap: 2024-09-17T15:52:26.888 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:52:26.888 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.888 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:52:26.887108+0000 2024-09-17T15:52:26.888 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:52:26.887108+0000 2024-09-17T15:52:26.888 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.888 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.888 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:52:26.889 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:52:26.889 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:52:26.889 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:52:26.889 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:52:26.889 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:52:26.889 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:52:26.889 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:52:26.889 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:52:26.889 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f6046241640 1 Processor -- start 2024-09-17T15:52:26.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f6046241640 1 -- start start 2024-09-17T15:52:26.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f6046241640 10 monclient: init 2024-09-17T15:52:26.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f6046241640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f6046241640 10 monclient: _add_conns ranks=[6,5,0,2,8,1,4,3,7] 2024-09-17T15:52:26.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f6046241640 10 monclient(hunting): picked mon.noname-d con 0x7f60400a54b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:52:26.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f6046241640 10 monclient(hunting): picked mon.noname-h con 0x7f6040135c80 addr v1:172.21.15.100:6791/0 2024-09-17T15:52:26.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f6046241640 10 monclient(hunting): picked mon.noname-a con 0x7f60401360a0 addr v1:172.21.15.64:6789/0 2024-09-17T15:52:26.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f6046241640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60400aacb0 con 0x7f60401360a0 2024-09-17T15:52:26.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f6046241640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f604009d780 con 0x7f6040135c80 2024-09-17T15:52:26.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f603f7fe640 1 -- >> v1:172.21.15.64:6792/0 conn(0x7f60400a54b0 legacy=0x7f60400a5890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6792/0 2024-09-17T15:52:26.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f6046241640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f604009da60 con 0x7f60400a54b0 2024-09-17T15:52:26.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f6046241640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f6046241640 10 monclient(hunting): authenticate will time out at 1144.562735s 2024-09-17T15:52:26.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f603effd640 1 -- >> v1:172.21.15.100:6791/0 conn(0x7f6040135c80 legacy=0x7f604013df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.100:6791/0 2024-09-17T15:52:26.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f603ffff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f60401360a0 0x7f6040141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:52626/0 (socket says 172.21.15.64:52626) 2024-09-17T15:52:26.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.887+0000 7f603ffff640 1 -- 172.21.15.64:0/28664724 learned_addr learned my addr 172.21.15.64:0/28664724 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:52:26.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 1 -- 172.21.15.64:0/28664724 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 420225766 0 0) 0x7f60400aacb0 con 0x7f60401360a0 2024-09-17T15:52:26.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient(hunting): my global_id is 124920 2024-09-17T15:52:26.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 1 -- 172.21.15.64:0/28664724 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6028003520 con 0x7f60401360a0 2024-09-17T15:52:26.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 1 -- 172.21.15.64:0/28664724 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 416425691 0 0) 0x7f6028003520 con 0x7f60401360a0 2024-09-17T15:52:26.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 1 -- 172.21.15.64:0/28664724 >> v1:172.21.15.64:6792/0 conn(0x7f60400a54b0 legacy=0x7f60400a5890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:26.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 1 -- 172.21.15.64:0/28664724 >> v1:172.21.15.100:6791/0 conn(0x7f6040135c80 legacy=0x7f604013df10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:26.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 1 monclient: found mon.noname-a 2024-09-17T15:52:26.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:52:26.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 1 -- 172.21.15.64:0/28664724 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604009dd40 con 0x7f60401360a0 2024-09-17T15:52:26.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 1 -- 172.21.15.64:0/28664724 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6038001bb0 con 0x7f60401360a0 2024-09-17T15:52:26.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f6046241640 5 monclient: authenticate success, global_id 124920 2024-09-17T15:52:26.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f6046241640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:26.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:26.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient: dump: 2024-09-17T15:52:26.902 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f6046241640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 1 -- 172.21.15.64:0/28664724 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6038001dd0 con 0x7f60401360a0 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 1 -- 172.21.15.64:0/28664724 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6038004f60 con 0x7f60401360a0 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f6046241640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f6046241640 4 set_mon_vals no callback set 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f603e7fc640 10 monclient: dump: 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.904 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.905 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.905 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.905 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.905 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.905 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.905 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.905 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.905 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.905 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.905 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.905 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.905 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f6046241640 10 monclient: shutdown 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f6046241640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f6046241640 1 -- 172.21.15.64:0/28664724 >> v1:172.21.15.64:6789/0 conn(0x7f60401360a0 legacy=0x7f6040141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f6046241640 1 -- 172.21.15.64:0/28664724 shutdown_connections 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.888+0000 7f6046241640 1 -- 172.21.15.64:0/28664724 >> 172.21.15.64:0/28664724 conn(0x7f604006daa0 msgr2=0x7f60400ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.889+0000 7f6046241640 1 -- 172.21.15.64:0/28664724 shutdown_connections 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.889+0000 7f6046241640 1 -- 172.21.15.64:0/28664724 wait complete. 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.889+0000 7f6046241640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.889+0000 7f6046241640 10 monclient: monmap: 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:52:26.906 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.907 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:52:26.907 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:52:26.907 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:52:26.907 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:52:26.907 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:52:26.907 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:52:26.907 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:52:26.907 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:52:26.907 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:52:26.907 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:52:26.907 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.889+0000 7f6046241640 1 Processor -- start 2024-09-17T15:52:26.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6046241640 1 -- start start 2024-09-17T15:52:26.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6046241640 10 monclient: init 2024-09-17T15:52:26.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6046241640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6046241640 10 monclient: _add_conns ranks=[2,4,8,7,5,3,1,0,6] 2024-09-17T15:52:26.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6046241640 10 monclient(hunting): picked mon.noname-b con 0x7f60400a54b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:52:26.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6046241640 10 monclient(hunting): picked mon.noname-c con 0x7f6040135c80 addr v1:172.21.15.64:6791/0 2024-09-17T15:52:26.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6046241640 10 monclient(hunting): picked mon.noname-e con 0x7f60401360a0 addr v1:172.21.15.64:6793/0 2024-09-17T15:52:26.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6046241640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6040135000 con 0x7f60400a54b0 2024-09-17T15:52:26.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6046241640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60400a1580 con 0x7f6040135c80 2024-09-17T15:52:26.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6046241640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60400a08b0 con 0x7f60401360a0 2024-09-17T15:52:26.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6046241640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f603ffff640 1 -- >> v1:172.21.15.64:6793/0 conn(0x7f60401360a0 legacy=0x7f60401d5e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.64:6793/0 2024-09-17T15:52:26.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6046241640 10 monclient(hunting): authenticate will time out at 1144.565544s 2024-09-17T15:52:26.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f603f7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f60400a54b0 0x7f604009fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:41290/0 (socket says 172.21.15.64:41290) 2024-09-17T15:52:26.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f603f7fe640 1 -- 172.21.15.64:0/982685266 learned_addr learned my addr 172.21.15.64:0/982685266 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:52:26.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4242540209 0 0) 0x7f60400a1580 con 0x7f6040135c80 2024-09-17T15:52:26.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6027fff640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6027fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6027fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6027fff640 10 monclient(hunting): my global_id is 154363 2024-09-17T15:52:26.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f602c003520 con 0x7f6040135c80 2024-09-17T15:52:26.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3638154124 0 0) 0x7f6040135000 con 0x7f60400a54b0 2024-09-17T15:52:26.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6027fff640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6027fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6027fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6027fff640 10 monclient(hunting): my global_id is 125021 2024-09-17T15:52:26.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.890+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60400a1580 con 0x7f60400a54b0 2024-09-17T15:52:26.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2996009817 0 0) 0x7f602c003520 con 0x7f6040135c80 2024-09-17T15:52:26.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6040135000 con 0x7f6040135c80 2024-09-17T15:52:26.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 715091737 0 0) 0x7f60400a1580 con 0x7f60400a54b0 2024-09-17T15:52:26.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f602c003520 con 0x7f60400a54b0 2024-09-17T15:52:26.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6034004360 con 0x7f6040135c80 2024-09-17T15:52:26.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:52:26.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:26.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient(hunting): dump: 2024-09-17T15:52:26.910 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.910 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.910 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.910 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.911 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.911 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.911 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.911 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.911 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.911 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.911 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.911 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.911 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.911 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.911 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.911 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:52:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f60300042b0 con 0x7f60400a54b0 2024-09-17T15:52:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:52:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:52:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient(hunting): dump: 2024-09-17T15:52:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.912 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.913 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4072180288 0 0) 0x7f6040135000 con 0x7f6040135c80 2024-09-17T15:52:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 >> v1:172.21.15.64:6793/0 conn(0x7f60401360a0 legacy=0x7f60401d5e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:52:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 >> v1:172.21.15.64:6790/0 conn(0x7f60400a54b0 legacy=0x7f604009fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 1 monclient: found mon.c 2024-09-17T15:52:26.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:26.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60400a38e0 con 0x7f6040135c80 2024-09-17T15:52:26.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6046241640 5 monclient: authenticate success, global_id 154363 2024-09-17T15:52:26.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6046241640 10 monclient: _renew_subs 2024-09-17T15:52:26.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6046241640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:26.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6046241640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f60400a3040 con 0x7f6040135c80 2024-09-17T15:52:26.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6046241640 10 monclient: _renew_subs 2024-09-17T15:52:26.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6046241640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:52:26.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6046241640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f60400a2380 con 0x7f6040135c80 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6034004500 con 0x7f6040135c80 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6034004a70 con 0x7f6040135c80 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f603dffb640 4 set_mon_vals no callback set 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.891+0000 7f6027fff640 10 monclient: dump: 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.892+0000 7f6046241640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.892+0000 7f6046241640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f60200065c0 con 0x7f6020002860 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.892+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f6034004c10 con 0x7f6040135c80 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.892+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(291..291 src has 192..291) ==== 3898+0+0 (unknown 1672347257 0 0) 0x7f603404ec00 con 0x7f6040135c80 2024-09-17T15:52:26.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.892+0000 7f603ffff640 10 monclient: get_auth_request con 0x7f602c0408b0 auth_method 0 2024-09-17T15:52:26.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.892+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4286698260 0 0) 0x7f60200065c0 con 0x7f6020002860 2024-09-17T15:52:26.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.892+0000 7f6027fff640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.892+0000 7f6027fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.892+0000 7f6027fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.892+0000 7f6027fff640 10 monclient(hunting): my global_id is 115662 2024-09-17T15:52:26.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.892+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6040135000 con 0x7f6020002860 2024-09-17T15:52:26.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.893+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3782259178 0 0) 0x7f6040135000 con 0x7f6020002860 2024-09-17T15:52:26.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.893+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60200065c0 con 0x7f6020002860 2024-09-17T15:52:26.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.893+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6030004b50 con 0x7f6020002860 2024-09-17T15:52:26.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.893+0000 7f6027fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.893+0000 7f6027fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:52:26.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.893+0000 7f6027fff640 10 monclient: dump: 2024-09-17T15:52:26.917 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.919 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.893+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2262440935 0 0) 0x7f60200065c0 con 0x7f6020002860 2024-09-17T15:52:26.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.893+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.100:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f602000a160 con 0x7f6020002860 2024-09-17T15:52:26.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.893+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6030004ef0 con 0x7f6020002860 2024-09-17T15:52:26.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.893+0000 7f6027fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.893+0000 7f6027fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:26.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.893+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 >> v1:172.21.15.100:6790/0 conn(0x7f6020002860 legacy=0x7f6020004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.893+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 >> v1:172.21.15.100:6790/0 conn(0x7f6020002860 legacy=0x7f6020004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:26.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.899+0000 7f6046241640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:52:26.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.899+0000 7f6046241640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6020009e50 con 0x7f6020009110 2024-09-17T15:52:26.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1578459050 0 0) 0x7f6020009e50 con 0x7f6020009110 2024-09-17T15:52:26.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 10 monclient(hunting): _negotiate 2024-09-17T15:52:26.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 10 monclient(hunting): my global_id is 115671 2024-09-17T15:52:26.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60200065c0 con 0x7f6020009110 2024-09-17T15:52:26.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3229972053 0 0) 0x7f60200065c0 con 0x7f6020009110 2024-09-17T15:52:26.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6020009e50 con 0x7f6020009110 2024-09-17T15:52:26.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6030005690 con 0x7f6020009110 2024-09-17T15:52:26.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:52:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 10 monclient: dump: 2024-09-17T15:52:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:52:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:52:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:52:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:52:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:52:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:52:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:52:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:52:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:52:26.922 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:52:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:52:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:52:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:52:26.923 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:52:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1605520679 0 0) 0x7f6020009e50 con 0x7f6020009110 2024-09-17T15:52:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.900+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 --> v1:172.21.15.100:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f602000c3f0 con 0x7f6020009110 2024-09-17T15:52:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7667 (unknown 0 0 4107943693) 0x7f6030006bc0 con 0x7f6020009110 2024-09-17T15:52:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f6027fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:52:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f6027fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 >> v1:172.21.15.100:6790/0 conn(0x7f6020009110 legacy=0x7f60200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f6027fff640 1 -- 172.21.15.64:0/982685266 >> v1:172.21.15.100:6790/0 conn(0x7f6020009110 legacy=0x7f60200095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:26.924 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:52:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:52:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:52:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:52:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 40, 2024-09-17T15:52:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 270124, 2024-09-17T15:52:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:52:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:52:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:52:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: 4 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 3, 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:52:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:52:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:52:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:52:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:52:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:52:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:52:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:52:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:52:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:52:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:52:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:52:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:52:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:52:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:52:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:52:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:52:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:52:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:52:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:52:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:52:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:52:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:52:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:52:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:52:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:52:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:52:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:52:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:52:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:52:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:52:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:52:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:52:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:52:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:52:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:52:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:52:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:52:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:52:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:52:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:52:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:52:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:52:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:52:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:52:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:52:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:52:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:52:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:52:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:52:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:52:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:52:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:52:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:52:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:52:26.943 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:52:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f6046241640 1 -- 172.21.15.64:0/982685266 >> v1:172.21.15.100:6812/4210963484 conn(0x7f602c0408b0 legacy=0x7f602c042d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f6046241640 10 monclient: shutdown 2024-09-17T15:52:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f6046241640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f6046241640 1 -- 172.21.15.64:0/982685266 >> v1:172.21.15.64:6791/0 conn(0x7f6040135c80 legacy=0x7f60400a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f603ffff640 1 -- 172.21.15.64:0/982685266 reap_dead start 2024-09-17T15:52:26.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f603ffff640 1 -- 172.21.15.64:0/982685266 reap_dead start 2024-09-17T15:52:26.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f6046241640 1 -- 172.21.15.64:0/982685266 shutdown_connections 2024-09-17T15:52:26.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f6046241640 1 -- 172.21.15.64:0/982685266 >> 172.21.15.64:0/982685266 conn(0x7f604006daa0 msgr2=0x7f60400a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.901+0000 7f6046241640 1 -- 172.21.15.64:0/982685266 shutdown_connections 2024-09-17T15:52:26.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:52:26.902+0000 7f6046241640 1 -- 172.21.15.64:0/982685266 wait complete. 2024-09-17T15:52:26.945 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-17T15:52:26.945 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-17T15:52:30.291 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39810]: starting. 2024-09-17T15:52:30.292 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39810]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39809 2024-09-17T15:52:30.292 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39810]: created object 0... 2024-09-17T15:52:30.292 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39810]: created object 25... 2024-09-17T15:52:30.292 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39810]: created object 49... 2024-09-17T15:52:30.292 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39810]: finishing. 2024-09-17T15:52:30.292 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39810]: shutting down. 2024-09-17T15:52:30.511 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39811]: starting. 2024-09-17T15:52:30.511 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39811]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39809 2024-09-17T15:52:30.511 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39811]: shutting down. 2024-09-17T15:52:32.620 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:32.620 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39850]: starting. 2024-09-17T15:52:32.620 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39850]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39809 2024-09-17T15:52:32.621 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39850]: created object 0... 2024-09-17T15:52:32.621 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39850]: created object 25... 2024-09-17T15:52:32.621 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39850]: created object 49... 2024-09-17T15:52:32.621 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39850]: finishing. 2024-09-17T15:52:32.621 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39850]: shutting down. 2024-09-17T15:52:32.631 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:32.631 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39852]: starting. 2024-09-17T15:52:32.631 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39852]: listing objects. 2024-09-17T15:52:32.631 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39852]: listed object 0... 2024-09-17T15:52:32.631 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39852]: listed object 25... 2024-09-17T15:52:32.631 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39852]: saw 50 objects 2024-09-17T15:52:32.631 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39852]: shutting down. 2024-09-17T15:52:33.539 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:33.539 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39851]: starting. 2024-09-17T15:52:33.539 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39851]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39809 2024-09-17T15:52:33.539 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39851]: shutting down. 2024-09-17T15:52:33.540 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:33.540 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:33.540 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:52:33.543 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39921]: starting. 2024-09-17T15:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39921]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39920 2024-09-17T15:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39921]: created object 0... 2024-09-17T15:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39921]: created object 25... 2024-09-17T15:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39921]: created object 49... 2024-09-17T15:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39921]: finishing. 2024-09-17T15:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[39921]: shutting down. 2024-09-17T15:52:39.572 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39922]: starting. 2024-09-17T15:52:39.572 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39922]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39920 2024-09-17T15:52:39.572 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[39922]: shutting down. 2024-09-17T15:52:41.690 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:41.690 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39957]: starting. 2024-09-17T15:52:41.690 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39957]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-39920 2024-09-17T15:52:41.690 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39957]: created object 0... 2024-09-17T15:52:41.690 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39957]: created object 25... 2024-09-17T15:52:41.690 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39957]: created object 49... 2024-09-17T15:52:41.691 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39957]: finishing. 2024-09-17T15:52:41.691 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[39957]: shutting down. 2024-09-17T15:52:41.700 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:41.701 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39959]: starting. 2024-09-17T15:52:41.701 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39959]: listing objects. 2024-09-17T15:52:41.701 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39959]: listed object 0... 2024-09-17T15:52:41.701 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39959]: listed object 25... 2024-09-17T15:52:41.701 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39959]: saw 50 objects 2024-09-17T15:52:41.701 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[39959]: shutting down. 2024-09-17T15:52:42.607 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:42.607 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39958]: starting. 2024-09-17T15:52:42.608 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39958]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-39920 2024-09-17T15:52:42.608 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[39958]: shutting down. 2024-09-17T15:52:42.609 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:42.609 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:42.609 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:52:42.612 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:52:44.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40028]: starting. 2024-09-17T15:52:44.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40028]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-40027 2024-09-17T15:52:44.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40028]: created object 0... 2024-09-17T15:52:44.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40028]: created object 25... 2024-09-17T15:52:44.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40028]: created object 49... 2024-09-17T15:52:44.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40028]: finishing. 2024-09-17T15:52:44.532 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40028]: shutting down. 2024-09-17T15:52:45.455 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[40029]: starting. 2024-09-17T15:52:45.456 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[40029]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-40027 2024-09-17T15:52:45.456 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[40029]: shutting down. 2024-09-17T15:52:47.562 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:47.562 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40064]: starting. 2024-09-17T15:52:47.562 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40064]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-40027 2024-09-17T15:52:47.562 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40064]: created object 0... 2024-09-17T15:52:47.562 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40064]: created object 25... 2024-09-17T15:52:47.562 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40064]: created object 49... 2024-09-17T15:52:47.562 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40064]: finishing. 2024-09-17T15:52:47.562 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40064]: shutting down. 2024-09-17T15:52:47.572 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:47.572 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40066]: starting. 2024-09-17T15:52:47.572 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40066]: listing objects. 2024-09-17T15:52:47.572 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40066]: listed object 0... 2024-09-17T15:52:47.572 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40066]: listed object 25... 2024-09-17T15:52:47.573 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40066]: saw 50 objects 2024-09-17T15:52:47.573 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40066]: shutting down. 2024-09-17T15:52:48.479 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:48.480 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[40065]: starting. 2024-09-17T15:52:48.480 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[40065]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-40027 2024-09-17T15:52:48.480 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[40065]: shutting down. 2024-09-17T15:52:48.481 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:48.481 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:48.481 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:52:48.484 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:52:50.573 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40135]: starting. 2024-09-17T15:52:50.573 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40135]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-40134 2024-09-17T15:52:50.573 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40135]: created object 0... 2024-09-17T15:52:50.573 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40135]: created object 25... 2024-09-17T15:52:50.573 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40135]: created object 49... 2024-09-17T15:52:50.573 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40135]: finishing. 2024-09-17T15:52:50.573 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40135]: shutting down. 2024-09-17T15:52:51.495 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[40136]: starting. 2024-09-17T15:52:51.495 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[40136]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-40134 2024-09-17T15:52:51.496 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[40136]: shutting down. 2024-09-17T15:52:53.528 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:53.528 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40171]: starting. 2024-09-17T15:52:53.528 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40171]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-40134 2024-09-17T15:52:53.528 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40171]: created object 0... 2024-09-17T15:52:53.528 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40171]: created object 25... 2024-09-17T15:52:53.528 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40171]: created object 49... 2024-09-17T15:52:53.528 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40171]: finishing. 2024-09-17T15:52:53.528 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40171]: shutting down. 2024-09-17T15:52:53.539 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:53.539 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40173]: starting. 2024-09-17T15:52:53.539 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40173]: listing objects. 2024-09-17T15:52:53.539 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40173]: listed object 0... 2024-09-17T15:52:53.539 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40173]: listed object 25... 2024-09-17T15:52:53.539 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40173]: saw 50 objects 2024-09-17T15:52:53.539 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40173]: shutting down. 2024-09-17T15:52:54.463 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:54.463 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[40172]: starting. 2024-09-17T15:52:54.463 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[40172]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-40134 2024-09-17T15:52:54.463 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[40172]: shutting down. 2024-09-17T15:52:54.464 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:54.464 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:54.464 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:52:54.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:52:56.599 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40242]: starting. 2024-09-17T15:52:56.599 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40242]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-40241 2024-09-17T15:52:56.599 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40242]: created object 0... 2024-09-17T15:52:56.599 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40242]: created object 25... 2024-09-17T15:52:56.599 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40242]: created object 49... 2024-09-17T15:52:56.599 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40242]: finishing. 2024-09-17T15:52:56.599 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40242]: shutting down. 2024-09-17T15:52:57.520 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[40243]: starting. 2024-09-17T15:52:57.520 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[40243]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-40241 2024-09-17T15:52:57.521 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[40243]: shutting down. 2024-09-17T15:52:59.612 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:59.612 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40278]: starting. 2024-09-17T15:52:59.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40278]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-40241 2024-09-17T15:52:59.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40278]: created object 0... 2024-09-17T15:52:59.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40278]: created object 25... 2024-09-17T15:52:59.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40278]: created object 49... 2024-09-17T15:52:59.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40278]: finishing. 2024-09-17T15:52:59.613 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40278]: shutting down. 2024-09-17T15:52:59.623 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:52:59.623 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40280]: starting. 2024-09-17T15:52:59.623 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40280]: listing objects. 2024-09-17T15:52:59.624 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40280]: listed object 0... 2024-09-17T15:52:59.624 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40280]: listed object 25... 2024-09-17T15:52:59.624 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40280]: saw 50 objects 2024-09-17T15:52:59.624 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40280]: shutting down. 2024-09-17T15:53:00.533 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:53:00.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[40279]: starting. 2024-09-17T15:53:00.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[40279]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-40241 2024-09-17T15:53:00.533 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[40279]: shutting down. 2024-09-17T15:53:00.537 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:53:00.537 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:53:00.537 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:53:00.539 DEBUG:teuthology.orchestra.run.smithi100:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-17T15:53:02.630 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40352]: starting. 2024-09-17T15:53:02.630 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40352]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-40351 2024-09-17T15:53:02.630 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40352]: created object 0... 2024-09-17T15:53:02.630 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40352]: created object 25... 2024-09-17T15:53:02.630 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40352]: created object 49... 2024-09-17T15:53:02.630 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40352]: finishing. 2024-09-17T15:53:02.631 INFO:teuthology.orchestra.run.smithi100.stdout:process_1_[40352]: shutting down. 2024-09-17T15:53:03.561 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[40353]: starting. 2024-09-17T15:53:03.561 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[40353]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-40351 2024-09-17T15:53:03.561 INFO:teuthology.orchestra.run.smithi100.stdout:process_2_[40353]: shutting down. 2024-09-17T15:53:05.685 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:53:05.685 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40389]: starting. 2024-09-17T15:53:05.685 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40389]: creating pool ceph_test_rados_delete_pools_parallel.smithi100-40351 2024-09-17T15:53:05.685 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40389]: created object 0... 2024-09-17T15:53:05.685 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40389]: created object 25... 2024-09-17T15:53:05.685 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40389]: created object 49... 2024-09-17T15:53:05.685 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40389]: finishing. 2024-09-17T15:53:05.685 INFO:teuthology.orchestra.run.smithi100.stdout:process_3_[40389]: shutting down. 2024-09-17T15:53:05.695 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:53:05.696 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40391]: starting. 2024-09-17T15:53:05.696 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40391]: listing objects. 2024-09-17T15:53:05.696 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40391]: listed object 0... 2024-09-17T15:53:05.696 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40391]: listed object 25... 2024-09-17T15:53:05.696 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40391]: saw 50 objects 2024-09-17T15:53:05.696 INFO:teuthology.orchestra.run.smithi100.stdout:process_5_[40391]: shutting down. 2024-09-17T15:53:06.601 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:53:06.601 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[40390]: starting. 2024-09-17T15:53:06.601 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[40390]: deleting pool ceph_test_rados_delete_pools_parallel.smithi100-40351 2024-09-17T15:53:06.601 INFO:teuthology.orchestra.run.smithi100.stdout:process_4_[40390]: shutting down. 2024-09-17T15:53:06.602 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:53:06.602 INFO:teuthology.orchestra.run.smithi100.stdout:******************************* 2024-09-17T15:53:06.602 INFO:teuthology.orchestra.run.smithi100.stdout:******* SUCCESS ********** 2024-09-17T15:53:06.605 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-17T15:53:06.614 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-17T15:53:56.946 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-09-17T15:53:56.947 INFO:tasks.ceph.mon.d:Restarting daemon 2024-09-17T15:53:56.947 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-09-17T15:53:56.951 INFO:tasks.ceph.mon.d:Started 2024-09-17T15:53:56.951 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.i 2024-09-17T15:53:56.951 INFO:tasks.ceph.mon.i:Restarting daemon 2024-09-17T15:53:56.952 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-09-17T15:53:56.955 INFO:tasks.ceph.mon.i:Started 2024-09-17T15:53:56.955 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-09-17T15:53:56.955 INFO:tasks.ceph.mon.e:Restarting daemon 2024-09-17T15:53:56.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-09-17T15:53:56.958 INFO:tasks.ceph.mon.e:Started 2024-09-17T15:53:56.958 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.h 2024-09-17T15:53:56.958 INFO:tasks.ceph.mon.h:Restarting daemon 2024-09-17T15:53:56.958 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-09-17T15:53:56.961 INFO:tasks.ceph.mon.h:Started 2024-09-17T15:53:56.961 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-17T15:53:56.962 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-17T15:53:56.962 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T15:54:11.963 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-17T15:54:11.963 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-17T15:54:11.964 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-17T15:54:11.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:54:12.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.083+0000 7fae9fe16640 10 monclient: get_monmap_and_config 2024-09-17T15:54:12.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.083+0000 7fae9fe16640 10 monclient: build_initial_monmap 2024-09-17T15:54:12.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.083+0000 7fae9fe16640 10 monclient: monmap: 2024-09-17T15:54:12.085 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:12.086 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.086 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:12.084542+0000 2024-09-17T15:54:12.086 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:12.084542+0000 2024-09-17T15:54:12.086 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:12.086 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.086 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:12.086 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:12.086 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:12.086 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:12.086 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:12.086 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:12.086 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:12.086 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:12.087 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:12.087 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.084+0000 7fae9fe16640 1 Processor -- start 2024-09-17T15:54:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.084+0000 7fae9fe16640 1 -- start start 2024-09-17T15:54:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.084+0000 7fae9fe16640 10 monclient: init 2024-09-17T15:54:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.084+0000 7fae9fe16640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.084+0000 7fae9fe16640 10 monclient: _add_conns ranks=[6,8,4,2,1,3,5,7,0] 2024-09-17T15:54:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.084+0000 7fae9fe16640 10 monclient(hunting): picked mon.noname-d con 0x7fae980a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9fe16640 10 monclient(hunting): picked mon.noname-e con 0x7fae98135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9fe16640 10 monclient(hunting): picked mon.noname-c con 0x7fae98135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9fe16640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae9809d6f0 con 0x7fae98135f70 2024-09-17T15:54:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9fe16640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae9809d9d0 con 0x7fae980a5520 2024-09-17T15:54:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9fe16640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae9809dcb0 con 0x7fae98135b50 2024-09-17T15:54:12.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9fe16640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:12.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9fe16640 10 monclient(hunting): authenticate will time out at 1249.760068s 2024-09-17T15:54:12.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9dbb2640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fae980a5520 0x7fae980a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:37152/0 (socket says 172.21.15.64:37152) 2024-09-17T15:54:12.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9d3b1640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fae98135b50 0x7fae9813deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51266/0 (socket says 172.21.15.64:51266) 2024-09-17T15:54:12.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9dbb2640 1 -- 172.21.15.64:0/3103513505 learned_addr learned my addr 172.21.15.64:0/3103513505 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:12.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 -- 172.21.15.64:0/3103513505 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2173061277 0 0) 0x7fae9809d9d0 con 0x7fae980a5520 2024-09-17T15:54:12.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient(hunting): my global_id is 174111 2024-09-17T15:54:12.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 -- 172.21.15.64:0/3103513505 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae84003520 con 0x7fae980a5520 2024-09-17T15:54:12.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 -- 172.21.15.64:0/3103513505 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 187761301 0 0) 0x7fae9809dcb0 con 0x7fae98135b50 2024-09-17T15:54:12.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient(hunting): my global_id is 174113 2024-09-17T15:54:12.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 -- 172.21.15.64:0/3103513505 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae9809d9d0 con 0x7fae98135b50 2024-09-17T15:54:12.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 -- 172.21.15.64:0/3103513505 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1619817519 0 0) 0x7fae9809d6f0 con 0x7fae98135f70 2024-09-17T15:54:12.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient(hunting): my global_id is 154615 2024-09-17T15:54:12.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 -- 172.21.15.64:0/3103513505 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae9809dcb0 con 0x7fae98135f70 2024-09-17T15:54:12.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 -- 172.21.15.64:0/3103513505 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 668750625 0 0) 0x7fae84003520 con 0x7fae980a5520 2024-09-17T15:54:12.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 -- 172.21.15.64:0/3103513505 >> v1:172.21.15.64:6793/0 conn(0x7fae98135b50 legacy=0x7fae9813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 -- 172.21.15.64:0/3103513505 >> v1:172.21.15.64:6791/0 conn(0x7fae98135f70 legacy=0x7fae981415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient: _finish_hunting 0 2024-09-17T15:54:12.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 monclient: found mon.noname-d 2024-09-17T15:54:12.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:12.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:54:12.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 -- 172.21.15.64:0/3103513505 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9809ccd0 con 0x7fae980a5520 2024-09-17T15:54:12.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient: _finish_auth 0 2024-09-17T15:54:12.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient: _check_auth_tickets 2024-09-17T15:54:12.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:12.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 -- 172.21.15.64:0/3103513505 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fae94001830 con 0x7fae980a5520 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient: dump: 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.099 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9fe16640 5 monclient: authenticate success, global_id 174111 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9fe16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 -- 172.21.15.64:0/3103513505 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fae940019d0 con 0x7fae980a5520 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9cbb0640 1 -- 172.21.15.64:0/3103513505 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fae94001cc0 con 0x7fae980a5520 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9fe16640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.085+0000 7fae9fe16640 4 set_mon_vals no callback set 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.086+0000 7fae9cbb0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.086+0000 7fae9cbb0640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.086+0000 7fae9cbb0640 10 monclient: dump: 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.102 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.102 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.102 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.086+0000 7fae9fe16640 10 monclient: shutdown 2024-09-17T15:54:12.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.086+0000 7fae9fe16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:12.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.086+0000 7fae9fe16640 1 -- 172.21.15.64:0/3103513505 >> v1:172.21.15.64:6792/0 conn(0x7fae980a5520 legacy=0x7fae980a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.086+0000 7fae9fe16640 1 -- 172.21.15.64:0/3103513505 shutdown_connections 2024-09-17T15:54:12.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.086+0000 7fae9fe16640 1 -- 172.21.15.64:0/3103513505 >> 172.21.15.64:0/3103513505 conn(0x7fae9806db30 msgr2=0x7fae980ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:12.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.086+0000 7fae9fe16640 1 -- 172.21.15.64:0/3103513505 shutdown_connections 2024-09-17T15:54:12.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.086+0000 7fae9fe16640 1 -- 172.21.15.64:0/3103513505 wait complete. 2024-09-17T15:54:12.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.087+0000 7fae9fe16640 10 monclient: build_initial_monmap 2024-09-17T15:54:12.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.087+0000 7fae9fe16640 10 monclient: monmap: 2024-09-17T15:54:12.102 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:12.103 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:12.104 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.087+0000 7fae9fe16640 1 Processor -- start 2024-09-17T15:54:12.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.087+0000 7fae9fe16640 1 -- start start 2024-09-17T15:54:12.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.087+0000 7fae9fe16640 10 monclient: init 2024-09-17T15:54:12.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae9fe16640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:12.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae9fe16640 10 monclient: _add_conns ranks=[4,6,2,3,1,5,8,7,0] 2024-09-17T15:54:12.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae9fe16640 10 monclient(hunting): picked mon.noname-c con 0x7fae980a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:12.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae9fe16640 10 monclient(hunting): picked mon.noname-d con 0x7fae98135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:12.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae9fe16640 10 monclient(hunting): picked mon.noname-b con 0x7fae98135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:12.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae9fe16640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae9809f380 con 0x7fae98135f70 2024-09-17T15:54:12.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae9fe16640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae980a8e40 con 0x7fae980a5520 2024-09-17T15:54:12.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae9fe16640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae980a0010 con 0x7fae98135b50 2024-09-17T15:54:12.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae9fe16640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:12.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae9fe16640 10 monclient(hunting): authenticate will time out at 1249.763225s 2024-09-17T15:54:12.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae9dbb2640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fae980a5520 0x7fae98131f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:48190/0 (socket says 172.21.15.64:48190) 2024-09-17T15:54:12.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae9d3b1640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fae98135b50 0x7fae98132690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:37166/0 (socket says 172.21.15.64:37166) 2024-09-17T15:54:12.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae9dbb2640 1 -- 172.21.15.64:0/2799647280 learned_addr learned my addr 172.21.15.64:0/2799647280 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:12.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3743202416 0 0) 0x7fae9809ccd0 con 0x7fae980a5520 2024-09-17T15:54:12.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): my global_id is 154624 2024-09-17T15:54:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae7c003520 con 0x7fae980a5520 2024-09-17T15:54:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3667235835 0 0) 0x7fae980a0010 con 0x7fae98135b50 2024-09-17T15:54:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): my global_id is 174120 2024-09-17T15:54:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae9809ccd0 con 0x7fae98135b50 2024-09-17T15:54:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1862112905 0 0) 0x7fae9809f380 con 0x7fae98135f70 2024-09-17T15:54:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): my global_id is 125183 2024-09-17T15:54:12.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae980a0010 con 0x7fae98135f70 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2606134725 0 0) 0x7fae7c003520 con 0x7fae980a5520 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae9809f380 con 0x7fae980a5520 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fae940017f0 con 0x7fae980a5520 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): dump: 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.107 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.108 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.108 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.108 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.108 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.108 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.108 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.108 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.108 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:12.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2652156099 0 0) 0x7fae9809ccd0 con 0x7fae98135b50 2024-09-17T15:54:12.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae7c003520 con 0x7fae98135b50 2024-09-17T15:54:12.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fae88001360 con 0x7fae98135b50 2024-09-17T15:54:12.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): dump: 2024-09-17T15:54:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.109 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.110 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.110 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.110 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.110 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.110 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4158664547 0 0) 0x7fae980a0010 con 0x7fae98135f70 2024-09-17T15:54:12.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fae9809ccd0 con 0x7fae98135f70 2024-09-17T15:54:12.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fae90001b50 con 0x7fae98135f70 2024-09-17T15:54:12.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:12.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:12.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.088+0000 7fae8e7fc640 10 monclient(hunting): dump: 2024-09-17T15:54:12.110 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.110 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.111 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1106387908 0 0) 0x7fae9809f380 con 0x7fae980a5520 2024-09-17T15:54:12.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 >> v1:172.21.15.64:6792/0 conn(0x7fae98135b50 legacy=0x7fae98132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 >> v1:172.21.15.64:6790/0 conn(0x7fae98135f70 legacy=0x7fae981d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:54:12.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 1 monclient: found mon.c 2024-09-17T15:54:12.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:12.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:12.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae98134a40 con 0x7fae980a5520 2024-09-17T15:54:12.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:54:12.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:54:12.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:12.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae9fe16640 5 monclient: authenticate success, global_id 154624 2024-09-17T15:54:12.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae9fe16640 10 monclient: _renew_subs 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae9fe16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae9fe16640 1 -- 172.21.15.64:0/2799647280 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fae9809f0b0 con 0x7fae980a5520 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae9fe16640 10 monclient: _renew_subs 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae9fe16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae9fe16640 1 -- 172.21.15.64:0/2799647280 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fae98132da0 con 0x7fae980a5520 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fae94001990 con 0x7fae980a5520 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fae94005270 con 0x7fae980a5520 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8ffff640 4 set_mon_vals no callback set 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.089+0000 7fae8e7fc640 10 monclient: dump: 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.113 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.114 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.090+0000 7fae9fe16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:12.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.090+0000 7fae9fe16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:12.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.090+0000 7fae9fe16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:12.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.090+0000 7fae9fe16640 1 -- 172.21.15.64:0/2799647280 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae80005080 con 0x7fae980a5520 2024-09-17T15:54:12.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.090+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fae94011b50 con 0x7fae980a5520 2024-09-17T15:54:12.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.090+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7fae9404ed40 con 0x7fae980a5520 2024-09-17T15:54:12.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.090+0000 7fae9d3b1640 10 monclient: get_auth_request con 0x7fae7c040c90 auth_method 0 2024-09-17T15:54:12.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.094+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fae980a8e40 con 0x7fae980a5520 2024-09-17T15:54:12.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.094+0000 7fae8e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:12.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.094+0000 7fae8e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:12.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.348+0000 7fae9fe16640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:54:12.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.348+0000 7fae9fe16640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:54:12.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.348+0000 7fae9fe16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:12.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.348+0000 7fae9fe16640 1 -- 172.21.15.64:0/2799647280 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fae80005370 con 0x7fae980a5520 2024-09-17T15:54:12.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.349+0000 7fae8e7fc640 1 -- 172.21.15.64:0/2799647280 <== mon.4 v1:172.21.15.64:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 352277721) 0x7fae94017b10 con 0x7fae980a5520 2024-09-17T15:54:12.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.349+0000 7fae8e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:54:12.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.349+0000 7fae8e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:12.351 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":46,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:54:12.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.351+0000 7fae9fe16640 1 -- 172.21.15.64:0/2799647280 >> v1:172.21.15.100:6812/4210963484 conn(0x7fae7c040c90 legacy=0x7fae7c043150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.351+0000 7fae9fe16640 10 monclient: shutdown 2024-09-17T15:54:12.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.351+0000 7fae9fe16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:12.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.351+0000 7fae9fe16640 1 -- 172.21.15.64:0/2799647280 >> v1:172.21.15.64:6791/0 conn(0x7fae980a5520 legacy=0x7fae98131f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.351+0000 7fae9fe16640 1 -- 172.21.15.64:0/2799647280 shutdown_connections 2024-09-17T15:54:12.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.351+0000 7fae9fe16640 1 -- 172.21.15.64:0/2799647280 >> 172.21.15.64:0/2799647280 conn(0x7fae9806db30 msgr2=0x7fae981334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:12.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.351+0000 7fae9fe16640 1 -- 172.21.15.64:0/2799647280 shutdown_connections 2024-09-17T15:54:12.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.351+0000 7fae9fe16640 1 -- 172.21.15.64:0/2799647280 wait complete. 2024-09-17T15:54:12.363 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-17T15:54:12.363 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-17T15:54:12.363 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:54:12.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.463+0000 7f7a20194640 10 monclient: get_monmap_and_config 2024-09-17T15:54:12.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.463+0000 7f7a20194640 10 monclient: build_initial_monmap 2024-09-17T15:54:12.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.463+0000 7f7a20194640 10 monclient: monmap: 2024-09-17T15:54:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:12.464360+0000 2024-09-17T15:54:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:12.464360+0000 2024-09-17T15:54:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:12.467 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a20194640 1 Processor -- start 2024-09-17T15:54:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a20194640 1 -- start start 2024-09-17T15:54:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a20194640 10 monclient: init 2024-09-17T15:54:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a20194640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:12.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a20194640 10 monclient: _add_conns ranks=[2,5,4,1,3,6,0,8,7] 2024-09-17T15:54:12.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a20194640 10 monclient(hunting): picked mon.noname-b con 0x7f7a180a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:12.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a20194640 10 monclient(hunting): picked mon.noname-h con 0x7f7a18135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:12.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a20194640 10 monclient(hunting): picked mon.noname-c con 0x7f7a18135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:12.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a20194640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a1809d6f0 con 0x7f7a180a5520 2024-09-17T15:54:12.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a20194640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a1809d9d0 con 0x7f7a18135f70 2024-09-17T15:54:12.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a20194640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a1809dcb0 con 0x7f7a18135b50 2024-09-17T15:54:12.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a20194640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:12.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a20194640 10 monclient(hunting): authenticate will time out at 1250.139887s 2024-09-17T15:54:12.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a1df30640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f7a180a5520 0x7f7a180a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45590/0 (socket says 172.21.15.64:45590) 2024-09-17T15:54:12.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.464+0000 7f7a1e731640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f7a18135f70 0x7f7a181415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:48192/0 (socket says 172.21.15.64:48192) 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1df30640 1 -- 172.21.15.64:0/2055159554 learned_addr learned my addr 172.21.15.64:0/2055159554 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 1 -- 172.21.15.64:0/2055159554 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2907127787 0 0) 0x7f7a1809d9d0 con 0x7f7a18135f70 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient(hunting): my global_id is 154633 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 1 -- 172.21.15.64:0/2055159554 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a04003520 con 0x7f7a18135f70 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 1 -- 172.21.15.64:0/2055159554 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 885560360 0 0) 0x7f7a1809d6f0 con 0x7f7a180a5520 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient(hunting): my global_id is 125192 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 1 -- 172.21.15.64:0/2055159554 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a1809d9d0 con 0x7f7a180a5520 2024-09-17T15:54:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 1 -- 172.21.15.64:0/2055159554 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 323112017 0 0) 0x7f7a1809d9d0 con 0x7f7a180a5520 2024-09-17T15:54:12.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 1 -- 172.21.15.64:0/2055159554 >> v1:172.21.15.100:6791/0 conn(0x7f7a18135b50 legacy=0x7f7a1813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 1 -- 172.21.15.64:0/2055159554 >> v1:172.21.15.64:6791/0 conn(0x7f7a18135f70 legacy=0x7f7a181415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient: _finish_hunting 0 2024-09-17T15:54:12.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 1 monclient: found mon.noname-b 2024-09-17T15:54:12.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:12.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:54:12.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 1 -- 172.21.15.64:0/2055159554 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1809ccd0 con 0x7f7a180a5520 2024-09-17T15:54:12.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient: _finish_auth 0 2024-09-17T15:54:12.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient: _check_auth_tickets 2024-09-17T15:54:12.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:12.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 1 -- 172.21.15.64:0/2055159554 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7a08001810 con 0x7f7a180a5520 2024-09-17T15:54:12.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a20194640 5 monclient: authenticate success, global_id 125192 2024-09-17T15:54:12.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a20194640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:12.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:54:12.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient: dump: 2024-09-17T15:54:12.475 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.475 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.475 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.475 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.475 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.475 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.475 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.476 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a20194640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 1 -- 172.21.15.64:0/2055159554 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7a080019b0 con 0x7f7a180a5520 2024-09-17T15:54:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:12.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 1 -- 172.21.15.64:0/2055159554 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7a08004be0 con 0x7f7a180a5520 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a20194640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a20194640 4 set_mon_vals no callback set 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.465+0000 7f7a1cf2e640 10 monclient: dump: 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.478 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.466+0000 7f7a20194640 10 monclient: shutdown 2024-09-17T15:54:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.466+0000 7f7a20194640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.466+0000 7f7a20194640 1 -- 172.21.15.64:0/2055159554 >> v1:172.21.15.64:6790/0 conn(0x7f7a180a5520 legacy=0x7f7a180a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.466+0000 7f7a20194640 1 -- 172.21.15.64:0/2055159554 shutdown_connections 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.466+0000 7f7a20194640 1 -- 172.21.15.64:0/2055159554 >> 172.21.15.64:0/2055159554 conn(0x7f7a1806db30 msgr2=0x7f7a180ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.466+0000 7f7a20194640 1 -- 172.21.15.64:0/2055159554 shutdown_connections 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.466+0000 7f7a20194640 1 -- 172.21.15.64:0/2055159554 wait complete. 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.467+0000 7f7a20194640 10 monclient: build_initial_monmap 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.467+0000 7f7a20194640 10 monclient: monmap: 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:12.479 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:12.480 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:12.480 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:12.480 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:12.480 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:12.480 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:12.480 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:12.480 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.467+0000 7f7a20194640 1 Processor -- start 2024-09-17T15:54:12.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.467+0000 7f7a20194640 1 -- start start 2024-09-17T15:54:12.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.467+0000 7f7a20194640 10 monclient: init 2024-09-17T15:54:12.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.467+0000 7f7a20194640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:12.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.467+0000 7f7a20194640 10 monclient: _add_conns ranks=[1,3,5,0,2,4,6,7,8] 2024-09-17T15:54:12.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.467+0000 7f7a20194640 10 monclient(hunting): picked mon.noname-f con 0x7f7a180a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:12.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.467+0000 7f7a20194640 10 monclient(hunting): picked mon.noname-g con 0x7f7a18135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:12.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.468+0000 7f7a20194640 10 monclient(hunting): picked mon.noname-h con 0x7f7a18135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:12.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.468+0000 7f7a20194640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a1809f540 con 0x7f7a180a5520 2024-09-17T15:54:12.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.468+0000 7f7a20194640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a180a0900 con 0x7f7a18135b50 2024-09-17T15:54:12.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.468+0000 7f7a20194640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a18134f30 con 0x7f7a18135f70 2024-09-17T15:54:12.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.468+0000 7f7a20194640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:12.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.468+0000 7f7a20194640 10 monclient(hunting): authenticate will time out at 1250.143084s 2024-09-17T15:54:12.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.468+0000 7f7a1df30640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f7a180a5520 0x7f7a181319e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.64:46092/0 (socket says 172.21.15.64:46092) 2024-09-17T15:54:12.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.468+0000 7f7a1df30640 1 -- 172.21.15.64:0/3641176256 learned_addr learned my addr 172.21.15.64:0/3641176256 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.472+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2382049695 0 0) 0x7f7a180a0900 con 0x7f7a18135b50 2024-09-17T15:54:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.472+0000 7f7a0e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.472+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.472+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.472+0000 7f7a0e7fc640 10 monclient(hunting): my global_id is 115878 2024-09-17T15:54:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.472+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79fc003520 con 0x7f7a18135b50 2024-09-17T15:54:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.472+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2142385327 0 0) 0x7f7a18134f30 con 0x7f7a18135f70 2024-09-17T15:54:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.472+0000 7f7a0e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.472+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.472+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.472+0000 7f7a0e7fc640 10 monclient(hunting): my global_id is 174119 2024-09-17T15:54:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.472+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a180a0900 con 0x7f7a18135f70 2024-09-17T15:54:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1687054746 0 0) 0x7f7a1809f540 con 0x7f7a180a5520 2024-09-17T15:54:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient(hunting): my global_id is 6895 2024-09-17T15:54:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a18134f30 con 0x7f7a180a5520 2024-09-17T15:54:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 158660522 0 0) 0x7f79fc003520 con 0x7f7a18135b50 2024-09-17T15:54:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a1809f540 con 0x7f7a18135b50 2024-09-17T15:54:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7a14001ad0 con 0x7f7a18135b50 2024-09-17T15:54:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:54:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient(hunting): dump: 2024-09-17T15:54:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.491 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3173849828 0 0) 0x7f7a180a0900 con 0x7f7a18135f70 2024-09-17T15:54:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79fc003520 con 0x7f7a18135f70 2024-09-17T15:54:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7a10001bc0 con 0x7f7a18135f70 2024-09-17T15:54:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:54:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient(hunting): dump: 2024-09-17T15:54:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.493 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.493 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.493 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.493 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.493 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.493 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4224542486 0 0) 0x7f7a1809f540 con 0x7f7a18135b50 2024-09-17T15:54:12.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 >> v1:172.21.15.100:6791/0 conn(0x7f7a18135f70 legacy=0x7f7a18132800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 >> v1:172.21.15.100:6789/0 conn(0x7f7a180a5520 legacy=0x7f7a181319e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:54:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 monclient: found mon.g 2024-09-17T15:54:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:54:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a180a0620 con 0x7f7a18135b50 2024-09-17T15:54:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:54:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:54:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a20194640 5 monclient: authenticate success, global_id 115878 2024-09-17T15:54:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a20194640 10 monclient: _renew_subs 2024-09-17T15:54:12.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a20194640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:54:12.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a20194640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7a1809f100 con 0x7f7a18135b50 2024-09-17T15:54:12.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a20194640 10 monclient: _renew_subs 2024-09-17T15:54:12.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a20194640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:54:12.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a20194640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7a181d7340 con 0x7f7a18135b50 2024-09-17T15:54:12.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.3 v1:172.21.15.100:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7a140038a0 con 0x7f7a18135b50 2024-09-17T15:54:12.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:12.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.3 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7a14003d90 con 0x7f7a18135b50 2024-09-17T15:54:12.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:54:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0e7fc640 10 monclient: dump: 2024-09-17T15:54:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.497 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.473+0000 7f7a0ffff640 4 set_mon_vals no callback set 2024-09-17T15:54:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.474+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.3 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f7a14010670 con 0x7f7a18135b50 2024-09-17T15:54:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.474+0000 7f7a20194640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:12.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.474+0000 7f7a20194640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7a180a8e30 con 0x7f7a18142f10 2024-09-17T15:54:12.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.474+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.3 v1:172.21.15.100:6790/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f7a1404d9a0 con 0x7f7a18135b50 2024-09-17T15:54:12.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.474+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 510393320 0 0) 0x7f7a180a8e30 con 0x7f7a18142f10 2024-09-17T15:54:12.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.474+0000 7f7a0e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.474+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.474+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.474+0000 7f7a0e7fc640 10 monclient(hunting): my global_id is 125082 2024-09-17T15:54:12.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a1809f540 con 0x7f7a18142f10 2024-09-17T15:54:12.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a1e731640 10 monclient: get_auth_request con 0x7f79fc040af0 auth_method 0 2024-09-17T15:54:12.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 497234389 0 0) 0x7f7a1809f540 con 0x7f7a18142f10 2024-09-17T15:54:12.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a180a8e30 con 0x7f7a18142f10 2024-09-17T15:54:12.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7a08001590 con 0x7f7a18142f10 2024-09-17T15:54:12.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:12.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 10 monclient: dump: 2024-09-17T15:54:12.500 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.500 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.500 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.500 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.500 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.500 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.500 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.500 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.500 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.500 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.500 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.500 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.500 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.501 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.501 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.501 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1261798951 0 0) 0x7f7a180a8e30 con 0x7f7a18142f10 2024-09-17T15:54:12.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.64:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7a180008d0 con 0x7f7a18142f10 2024-09-17T15:54:12.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7a080017b0 con 0x7f7a18142f10 2024-09-17T15:54:12.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:12.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:12.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 >> v1:172.21.15.64:6789/0 conn(0x7f7a18142f10 legacy=0x7f7a1812d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.475+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 >> v1:172.21.15.64:6789/0 conn(0x7f7a18142f10 legacy=0x7f7a1812d530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:12.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.481+0000 7f7a20194640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:54:12.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.481+0000 7f7a20194640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7a1802e830 con 0x7f7a1802e450 2024-09-17T15:54:12.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346496046 0 0) 0x7f7a1802e830 con 0x7f7a1802e450 2024-09-17T15:54:12.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 10 monclient(hunting): my global_id is 125091 2024-09-17T15:54:12.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a180a8e30 con 0x7f7a1802e450 2024-09-17T15:54:12.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1170847574 0 0) 0x7f7a180a8e30 con 0x7f7a1802e450 2024-09-17T15:54:12.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a1802e830 con 0x7f7a1802e450 2024-09-17T15:54:12.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f7a08004ad0 con 0x7f7a1802e450 2024-09-17T15:54:12.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 10 monclient: dump: 2024-09-17T15:54:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.503 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.504 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.504 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.504 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.504 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3228257122 0 0) 0x7f7a1802e830 con 0x7f7a1802e450 2024-09-17T15:54:12.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 --> v1:172.21.15.64:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7a1812e350 con 0x7f7a1802e450 2024-09-17T15:54:12.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 <== mon.0 v1:172.21.15.64:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 4263088366) 0x7f7a08005340 con 0x7f7a1802e450 2024-09-17T15:54:12.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:54:12.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:12.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 >> v1:172.21.15.64:6789/0 conn(0x7f7a1802e450 legacy=0x7f7a1812dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.482+0000 7f7a0e7fc640 1 -- 172.21.15.64:0/3641176256 >> v1:172.21.15.64:6789/0 conn(0x7f7a1802e450 legacy=0x7f7a1812dc40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:12.505 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:54:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:54:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:54:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 46, 2024-09-17T15:54:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 259400, 2024-09-17T15:54:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:54:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:54:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:54:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:54:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:54:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:54:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:54:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:54:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:54:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:54:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:54:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:54:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:54:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:54:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:54:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:54:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:54:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:54:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:54:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:54:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:54:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:54:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:54:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:54:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:54:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:54:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:54:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:54:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:54:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:54:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:54:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:54:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:54:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:54:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:54:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:54:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:54:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:54:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:54:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:54:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:54:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:54:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:54:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:54:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:54:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:54:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:54:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:54:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:54:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:54:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:54:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:54:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:54:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:54:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:54:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:54:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:54:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:54:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:54:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:54:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:54:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:54:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:54:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:54:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:54:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:54:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:54:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": [ 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:54:12.527 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:12.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.483+0000 7f7a20194640 1 -- 172.21.15.64:0/3641176256 >> v1:172.21.15.100:6812/4210963484 conn(0x7f79fc040af0 legacy=0x7f79fc042fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.483+0000 7f7a20194640 10 monclient: shutdown 2024-09-17T15:54:12.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.483+0000 7f7a20194640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:12.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.483+0000 7f7a20194640 1 -- 172.21.15.64:0/3641176256 >> v1:172.21.15.100:6790/0 conn(0x7f7a18135b50 legacy=0x7f7a181320f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.483+0000 7f7a1e731640 1 -- 172.21.15.64:0/3641176256 reap_dead start 2024-09-17T15:54:12.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.483+0000 7f7a1e731640 1 -- 172.21.15.64:0/3641176256 reap_dead start 2024-09-17T15:54:12.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.483+0000 7f7a20194640 1 -- 172.21.15.64:0/3641176256 shutdown_connections 2024-09-17T15:54:12.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.483+0000 7f7a20194640 1 -- 172.21.15.64:0/3641176256 >> 172.21.15.64:0/3641176256 conn(0x7f7a1806db30 msgr2=0x7f7a181334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:12.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.483+0000 7f7a20194640 1 -- 172.21.15.64:0/3641176256 shutdown_connections 2024-09-17T15:54:12.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.483+0000 7f7a20194640 1 -- 172.21.15.64:0/3641176256 wait complete. 2024-09-17T15:54:12.529 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:54:12.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-17T15:54:12.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:54:12.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.630+0000 7f53f77f8640 10 monclient: get_monmap_and_config 2024-09-17T15:54:12.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.630+0000 7f53f77f8640 10 monclient: build_initial_monmap 2024-09-17T15:54:12.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.630+0000 7f53f77f8640 10 monclient: monmap: 2024-09-17T15:54:12.632 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:12.632 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.632 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:12.631284+0000 2024-09-17T15:54:12.632 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:12.631284+0000 2024-09-17T15:54:12.633 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:12.633 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.633 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:12.633 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:12.633 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:12.633 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:12.633 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:12.633 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:12.633 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:12.633 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:12.633 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:12.634 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f77f8640 1 Processor -- start 2024-09-17T15:54:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f77f8640 1 -- start start 2024-09-17T15:54:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f77f8640 10 monclient: init 2024-09-17T15:54:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f77f8640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f77f8640 10 monclient: _add_conns ranks=[7,0,1,3,8,4,5,2,6] 2024-09-17T15:54:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f77f8640 10 monclient(hunting): picked mon.noname-i con 0x7f53f00a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f77f8640 10 monclient(hunting): picked mon.noname-a con 0x7f53f0135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f77f8640 10 monclient(hunting): picked mon.noname-f con 0x7f53f0135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f77f8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f009d6f0 con 0x7f53f0135b50 2024-09-17T15:54:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f77f8640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f009d9d0 con 0x7f53f0135f70 2024-09-17T15:54:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f77f8640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f009dcb0 con 0x7f53f00a5520 2024-09-17T15:54:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f77f8640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f77f8640 10 monclient(hunting): authenticate will time out at 1250.306749s 2024-09-17T15:54:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f4d93640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f53f0135b50 0x7f53f013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:34624/0 (socket says 172.21.15.64:34624) 2024-09-17T15:54:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.631+0000 7f53f4d93640 1 -- 172.21.15.64:0/353155199 learned_addr learned my addr 172.21.15.64:0/353155199 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 -- 172.21.15.64:0/353155199 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1602800669 0 0) 0x7f53f009d6f0 con 0x7f53f0135b50 2024-09-17T15:54:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient(hunting): my global_id is 125100 2024-09-17T15:54:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 -- 172.21.15.64:0/353155199 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53d8003520 con 0x7f53f0135b50 2024-09-17T15:54:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 -- 172.21.15.64:0/353155199 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 859202484 0 0) 0x7f53f009d9d0 con 0x7f53f0135f70 2024-09-17T15:54:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient(hunting): my global_id is 6904 2024-09-17T15:54:12.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 -- 172.21.15.64:0/353155199 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53f009d6f0 con 0x7f53f0135f70 2024-09-17T15:54:12.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 -- 172.21.15.64:0/353155199 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 451864181 0 0) 0x7f53f009dcb0 con 0x7f53f00a5520 2024-09-17T15:54:12.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient(hunting): my global_id is 174112 2024-09-17T15:54:12.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 -- 172.21.15.64:0/353155199 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53f009d9d0 con 0x7f53f00a5520 2024-09-17T15:54:12.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 -- 172.21.15.64:0/353155199 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 188350355 0 0) 0x7f53d8003520 con 0x7f53f0135b50 2024-09-17T15:54:12.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 -- 172.21.15.64:0/353155199 >> v1:172.21.15.100:6792/0 conn(0x7f53f00a5520 legacy=0x7f53f00a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 -- 172.21.15.64:0/353155199 >> v1:172.21.15.100:6789/0 conn(0x7f53f0135f70 legacy=0x7f53f01415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient: _finish_hunting 0 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 monclient: found mon.noname-a 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 -- 172.21.15.64:0/353155199 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f009ccd0 con 0x7f53f0135b50 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient: _finish_auth 0 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient: _check_auth_tickets 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53f77f8640 5 monclient: authenticate success, global_id 125100 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 -- 172.21.15.64:0/353155199 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f53e4001830 con 0x7f53f0135b50 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:54:12.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient: dump: 2024-09-17T15:54:12.652 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.652 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.652 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.652 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.652 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.652 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.652 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.652 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.652 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.652 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.652 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.652 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53f77f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 -- 172.21.15.64:0/353155199 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53e4001a50 con 0x7f53f0135b50 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 1 -- 172.21.15.64:0/353155199 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f53e4004be0 con 0x7f53f0135b50 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53f77f8640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53f77f8640 4 set_mon_vals no callback set 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53effff640 10 monclient: dump: 2024-09-17T15:54:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.654 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.655 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53f77f8640 10 monclient: shutdown 2024-09-17T15:54:12.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53f77f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:12.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.632+0000 7f53f77f8640 1 -- 172.21.15.64:0/353155199 >> v1:172.21.15.64:6789/0 conn(0x7f53f0135b50 legacy=0x7f53f013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.633+0000 7f53f77f8640 1 -- 172.21.15.64:0/353155199 shutdown_connections 2024-09-17T15:54:12.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.633+0000 7f53f77f8640 1 -- 172.21.15.64:0/353155199 >> 172.21.15.64:0/353155199 conn(0x7f53f006db30 msgr2=0x7f53f00ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:12.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.633+0000 7f53f77f8640 1 -- 172.21.15.64:0/353155199 shutdown_connections 2024-09-17T15:54:12.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.633+0000 7f53f77f8640 1 -- 172.21.15.64:0/353155199 wait complete. 2024-09-17T15:54:12.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.633+0000 7f53f77f8640 10 monclient: build_initial_monmap 2024-09-17T15:54:12.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.633+0000 7f53f77f8640 10 monclient: monmap: 2024-09-17T15:54:12.655 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:12.655 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:12.656 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:12.656 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:12.656 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:12.656 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:12.656 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:12.656 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:12.656 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:12.656 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:12.656 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:12.656 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:12.656 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:12.656 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:12.656 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:12.657 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f77f8640 1 Processor -- start 2024-09-17T15:54:12.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f77f8640 1 -- start start 2024-09-17T15:54:12.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f77f8640 10 monclient: init 2024-09-17T15:54:12.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f77f8640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:12.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f77f8640 10 monclient: _add_conns ranks=[7,3,5,1,8,0,2,4,6] 2024-09-17T15:54:12.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f77f8640 10 monclient(hunting): picked mon.noname-i con 0x7f53f00a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:12.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f77f8640 10 monclient(hunting): picked mon.noname-g con 0x7f53f0135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:12.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f77f8640 10 monclient(hunting): picked mon.noname-h con 0x7f53f0135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:12.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f77f8640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f0134840 con 0x7f53f0135b50 2024-09-17T15:54:12.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f77f8640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f0138a90 con 0x7f53f0135f70 2024-09-17T15:54:12.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f77f8640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f0137e10 con 0x7f53f00a5520 2024-09-17T15:54:12.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f77f8640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:12.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f77f8640 10 monclient(hunting): authenticate will time out at 1250.309536s 2024-09-17T15:54:12.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f4d93640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f53f0135b50 0x7f53f0137700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:46104/0 (socket says 172.21.15.64:46104) 2024-09-17T15:54:12.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f5594640 1 --1- >> v1:172.21.15.100:6792/0 conn(0x7f53f00a5520 0x7f53f0136ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6792/0 says I am v1:172.21.15.64:37720/0 (socket says 172.21.15.64:37720) 2024-09-17T15:54:12.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53f4d93640 1 -- 172.21.15.64:0/219852705 learned_addr learned my addr 172.21.15.64:0/219852705 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:12.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2736058956 0 0) 0x7f53f0134840 con 0x7f53f0135b50 2024-09-17T15:54:12.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53edffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53edffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.634+0000 7f53edffb640 10 monclient(hunting): my global_id is 115887 2024-09-17T15:54:12.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53dc003520 con 0x7f53f0135b50 2024-09-17T15:54:12.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 287167713 0 0) 0x7f53f0137e10 con 0x7f53f00a5520 2024-09-17T15:54:12.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): my global_id is 174121 2024-09-17T15:54:12.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53f0134840 con 0x7f53f00a5520 2024-09-17T15:54:12.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2131984038 0 0) 0x7f53f0138a90 con 0x7f53f0135f70 2024-09-17T15:54:12.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): my global_id is 174128 2024-09-17T15:54:12.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53f0137e10 con 0x7f53f0135f70 2024-09-17T15:54:12.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1626133718 0 0) 0x7f53f0134840 con 0x7f53f00a5520 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53f0138a90 con 0x7f53f00a5520 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f53e0001d80 con 0x7f53f00a5520 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): dump: 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.660 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 381021553 0 0) 0x7f53dc003520 con 0x7f53f0135b50 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53f0134840 con 0x7f53f0135b50 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2725648137 0 0) 0x7f53f0137e10 con 0x7f53f0135f70 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53dc003520 con 0x7f53f0135f70 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f53e4004b20 con 0x7f53f0135b50 2024-09-17T15:54:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:12.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:54:12.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): dump: 2024-09-17T15:54:12.662 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.662 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.662 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.662 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.662 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.662 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.662 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.662 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.662 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.662 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.662 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f53e80017d0 con 0x7f53f0135f70 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient(hunting): dump: 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.663 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1444195609 0 0) 0x7f53f0138a90 con 0x7f53f00a5520 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 >> v1:172.21.15.100:6791/0 conn(0x7f53f0135f70 legacy=0x7f53f01d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 >> v1:172.21.15.100:6790/0 conn(0x7f53f0135b50 legacy=0x7f53f0137700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient: _finish_hunting 0 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 monclient: found mon.i 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.100:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f013a550 con 0x7f53f00a5520 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient: _finish_auth 0 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient: _check_auth_tickets 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53f77f8640 5 monclient: authenticate success, global_id 174121 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53f77f8640 10 monclient: _renew_subs 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53f77f8640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53f77f8640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.100:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f53f013a830 con 0x7f53f00a5520 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.7 v1:172.21.15.100:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53e0002020 con 0x7f53f00a5520 2024-09-17T15:54:12.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.7 v1:172.21.15.100:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f53e0003f80 con 0x7f53f00a5520 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53edffb640 10 monclient: dump: 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.666 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53f77f8640 10 monclient: _renew_subs 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53f77f8640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.100:6792/0 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53f77f8640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.100:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f53f0142440 con 0x7f53f00a5520 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.635+0000 7f53ef7fe640 4 set_mon_vals no callback set 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.636+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.7 v1:172.21.15.100:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f53e0004120 con 0x7f53f00a5520 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.636+0000 7f53f77f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.636+0000 7f53f77f8640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f53f00008d0 con 0x7f53f0142f10 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.636+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.7 v1:172.21.15.100:6792/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f53e004ed30 con 0x7f53f00a5520 2024-09-17T15:54:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.636+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3265725907 0 0) 0x7f53f00008d0 con 0x7f53f0142f10 2024-09-17T15:54:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.636+0000 7f53edffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.636+0000 7f53edffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.636+0000 7f53edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.636+0000 7f53edffb640 10 monclient(hunting): my global_id is 125201 2024-09-17T15:54:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.636+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53f0138a90 con 0x7f53f0142f10 2024-09-17T15:54:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.636+0000 7f53f5d95640 10 monclient: get_auth_request con 0x7f53dc040d90 auth_method 0 2024-09-17T15:54:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.637+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1697101704 0 0) 0x7f53f0138a90 con 0x7f53f0142f10 2024-09-17T15:54:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.637+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53f00008d0 con 0x7f53f0142f10 2024-09-17T15:54:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.637+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f53e4001c20 con 0x7f53f0142f10 2024-09-17T15:54:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.637+0000 7f53edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.637+0000 7f53edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.637+0000 7f53edffb640 10 monclient: dump: 2024-09-17T15:54:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.669 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.637+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 15921546 0 0) 0x7f53f00008d0 con 0x7f53f0142f10 2024-09-17T15:54:12.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.637+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.64:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53f0064460 con 0x7f53f0142f10 2024-09-17T15:54:12.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.637+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f53e4001be0 con 0x7f53f0142f10 2024-09-17T15:54:12.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.637+0000 7f53edffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:12.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.637+0000 7f53edffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:12.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.637+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 >> v1:172.21.15.64:6790/0 conn(0x7f53f0142f10 legacy=0x7f53f0063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.637+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 >> v1:172.21.15.64:6790/0 conn(0x7f53f0142f10 legacy=0x7f53f0063d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:12.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.643+0000 7f53f77f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:54:12.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.643+0000 7f53f77f8640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f53f00a1ba0 con 0x7f53f002e5c0 2024-09-17T15:54:12.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4259198080 0 0) 0x7f53f00a1ba0 con 0x7f53f002e5c0 2024-09-17T15:54:12.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 10 monclient(hunting): my global_id is 125210 2024-09-17T15:54:12.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53f00008d0 con 0x7f53f002e5c0 2024-09-17T15:54:12.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3396660212 0 0) 0x7f53f00008d0 con 0x7f53f002e5c0 2024-09-17T15:54:12.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53f00a1ba0 con 0x7f53f002e5c0 2024-09-17T15:54:12.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f53e4005860 con 0x7f53f002e5c0 2024-09-17T15:54:12.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:12.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 10 monclient: dump: 2024-09-17T15:54:12.671 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.671 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.671 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.671 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.671 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3562677555 0 0) 0x7f53f00a1ba0 con 0x7f53f002e5c0 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 --> v1:172.21.15.64:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f53f002e9a0 con 0x7f53f002e5c0 2024-09-17T15:54:12.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 <== mon.2 v1:172.21.15.64:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1923796066) 0x7f53e4007da0 con 0x7f53f002e5c0 2024-09-17T15:54:12.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:54:12.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:12.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 >> v1:172.21.15.64:6790/0 conn(0x7f53f002e5c0 legacy=0x7f53f00a1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.644+0000 7f53edffb640 1 -- 172.21.15.64:0/219852705 >> v1:172.21.15.64:6790/0 conn(0x7f53f002e5c0 legacy=0x7f53f00a1490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:12.673 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:54:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:54:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:54:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 46, 2024-09-17T15:54:12.673 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 259562, 2024-09-17T15:54:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:54:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:54:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:54:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:54:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:54:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:54:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:54:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:54:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:54:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:54:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:54:12.674 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:54:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:54:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:12.675 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:54:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:54:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:12.676 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:54:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:54:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:54:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:54:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:54:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:54:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:54:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:54:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:54:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:54:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:54:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:54:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:54:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:54:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:12.678 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:54:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:54:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:54:12.679 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:54:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:54:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:54:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:54:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.680 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.682 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:54:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:54:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:54:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:54:12.683 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.684 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:54:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:54:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:54:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:54:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:54:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:54:12.686 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:54:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:54:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.688 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:54:12.689 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:54:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:54:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:54:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.690 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:54:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:54:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:54:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:54:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:54:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:54:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:54:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:12.692 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:54:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:54:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.693 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:54:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.694 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:54:12.694 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:12.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.645+0000 7f53f77f8640 1 -- 172.21.15.64:0/219852705 >> v1:172.21.15.100:6812/4210963484 conn(0x7f53dc040d90 legacy=0x7f53dc043250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.645+0000 7f53f77f8640 10 monclient: shutdown 2024-09-17T15:54:12.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.645+0000 7f53f77f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:12.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.645+0000 7f53f77f8640 1 -- 172.21.15.64:0/219852705 >> v1:172.21.15.100:6792/0 conn(0x7f53f00a5520 legacy=0x7f53f0136ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.645+0000 7f53f5d95640 1 -- 172.21.15.64:0/219852705 reap_dead start 2024-09-17T15:54:12.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.645+0000 7f53f5d95640 1 -- 172.21.15.64:0/219852705 reap_dead start 2024-09-17T15:54:12.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.645+0000 7f53f77f8640 1 -- 172.21.15.64:0/219852705 shutdown_connections 2024-09-17T15:54:12.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.645+0000 7f53f77f8640 1 -- 172.21.15.64:0/219852705 >> 172.21.15.64:0/219852705 conn(0x7f53f006db30 msgr2=0x7f53f00a4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:12.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.645+0000 7f53f77f8640 1 -- 172.21.15.64:0/219852705 shutdown_connections 2024-09-17T15:54:12.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.645+0000 7f53f77f8640 1 -- 172.21.15.64:0/219852705 wait complete. 2024-09-17T15:54:12.696 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:54:12.696 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-17T15:54:12.697 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:54:12.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.796+0000 7f2917e33640 10 monclient: get_monmap_and_config 2024-09-17T15:54:12.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.796+0000 7f2917e33640 10 monclient: build_initial_monmap 2024-09-17T15:54:12.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.796+0000 7f2917e33640 10 monclient: monmap: 2024-09-17T15:54:12.798 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:12.798 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:12.797611+0000 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:12.797611+0000 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:12.799 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.797+0000 7f2917e33640 1 Processor -- start 2024-09-17T15:54:12.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.797+0000 7f2917e33640 1 -- start start 2024-09-17T15:54:12.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.797+0000 7f2917e33640 10 monclient: init 2024-09-17T15:54:12.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:12.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 10 monclient: _add_conns ranks=[3,6,5,7,4,0,2,8,1] 2024-09-17T15:54:12.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 10 monclient(hunting): picked mon.noname-g con 0x7f29100a5520 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:12.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 10 monclient(hunting): picked mon.noname-d con 0x7f2910135b50 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 10 monclient(hunting): picked mon.noname-h con 0x7f2910135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291009d6f0 con 0x7f29100a5520 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291009d9d0 con 0x7f2910135f70 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291009dcb0 con 0x7f2910135b50 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 10 monclient(hunting): authenticate will time out at 1250.473161s 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f29153ce640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f2910135b50 0x7f291013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:37180/0 (socket says 172.21.15.64:37180) 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f29153ce640 1 -- 172.21.15.64:0/3525153378 learned_addr learned my addr 172.21.15.64:0/3525153378 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 -- 172.21.15.64:0/3525153378 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 262674710 0 0) 0x7f291009dcb0 con 0x7f2910135b50 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient(hunting): my global_id is 174129 2024-09-17T15:54:12.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 -- 172.21.15.64:0/3525153378 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28fc003520 con 0x7f2910135b50 2024-09-17T15:54:12.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 -- 172.21.15.64:0/3525153378 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4289711891 0 0) 0x7f291009d6f0 con 0x7f29100a5520 2024-09-17T15:54:12.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient(hunting): my global_id is 115896 2024-09-17T15:54:12.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 -- 172.21.15.64:0/3525153378 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f291009dcb0 con 0x7f29100a5520 2024-09-17T15:54:12.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 -- 172.21.15.64:0/3525153378 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1192198123 0 0) 0x7f291009d9d0 con 0x7f2910135f70 2024-09-17T15:54:12.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient(hunting): my global_id is 174137 2024-09-17T15:54:12.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 -- 172.21.15.64:0/3525153378 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f291009d6f0 con 0x7f2910135f70 2024-09-17T15:54:12.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 -- 172.21.15.64:0/3525153378 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3385240814 0 0) 0x7f28fc003520 con 0x7f2910135b50 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 -- 172.21.15.64:0/3525153378 >> v1:172.21.15.100:6791/0 conn(0x7f2910135f70 legacy=0x7f29101415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 -- 172.21.15.64:0/3525153378 >> v1:172.21.15.100:6790/0 conn(0x7f29100a5520 legacy=0x7f29100a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient: _finish_hunting 0 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 monclient: found mon.noname-d 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 -- 172.21.15.64:0/3525153378 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291009ccd0 con 0x7f2910135b50 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient: _finish_auth 0 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient: _check_auth_tickets 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 -- 172.21.15.64:0/3525153378 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2904001830 con 0x7f2910135b50 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 5 monclient: authenticate success, global_id 174129 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:54:12.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient: dump: 2024-09-17T15:54:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.812 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 -- 172.21.15.64:0/3525153378 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2904001a50 con 0x7f2910135b50 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 1 -- 172.21.15.64:0/3525153378 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2904004be0 con 0x7f2910135b50 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2917e33640 4 set_mon_vals no callback set 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.798+0000 7f2914bcd640 10 monclient: dump: 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.799+0000 7f2917e33640 10 monclient: shutdown 2024-09-17T15:54:12.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.799+0000 7f2917e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:12.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.799+0000 7f2917e33640 1 -- 172.21.15.64:0/3525153378 >> v1:172.21.15.64:6792/0 conn(0x7f2910135b50 legacy=0x7f291013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.799+0000 7f2917e33640 1 -- 172.21.15.64:0/3525153378 shutdown_connections 2024-09-17T15:54:12.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.799+0000 7f2917e33640 1 -- 172.21.15.64:0/3525153378 >> 172.21.15.64:0/3525153378 conn(0x7f291006db30 msgr2=0x7f29100ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:12.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.799+0000 7f2917e33640 1 -- 172.21.15.64:0/3525153378 shutdown_connections 2024-09-17T15:54:12.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.799+0000 7f2917e33640 1 -- 172.21.15.64:0/3525153378 wait complete. 2024-09-17T15:54:12.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.800+0000 7f2917e33640 10 monclient: build_initial_monmap 2024-09-17T15:54:12.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.800+0000 7f2917e33640 10 monclient: monmap: 2024-09-17T15:54:12.815 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:12.815 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:12.816 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:12.816 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:12.816 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:12.816 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:12.816 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:12.816 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:12.816 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:12.816 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:12.816 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:12.817 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:12.817 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:12.817 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:12.817 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:12.817 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.800+0000 7f2917e33640 1 Processor -- start 2024-09-17T15:54:12.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.800+0000 7f2917e33640 1 -- start start 2024-09-17T15:54:12.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.800+0000 7f2917e33640 10 monclient: init 2024-09-17T15:54:12.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.800+0000 7f2917e33640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:12.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.800+0000 7f2917e33640 10 monclient: _add_conns ranks=[6,5,8,1,3,4,0,7,2] 2024-09-17T15:54:12.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.800+0000 7f2917e33640 10 monclient(hunting): picked mon.noname-d con 0x7f29100a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:12.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.800+0000 7f2917e33640 10 monclient(hunting): picked mon.noname-h con 0x7f2910135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:12.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.800+0000 7f2917e33640 10 monclient(hunting): picked mon.noname-e con 0x7f2910135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.800+0000 7f2917e33640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29100a07b0 con 0x7f2910135b50 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f2917e33640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291009e470 con 0x7f29100a5520 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f2917e33640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2910134c90 con 0x7f2910135f70 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f2917e33640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f2917e33640 10 monclient(hunting): authenticate will time out at 1250.476066s 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f2915bcf640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f29100a5520 0x7f2910131850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:37192/0 (socket says 172.21.15.64:37192) 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f2915bcf640 1 -- 172.21.15.64:0/2983066128 learned_addr learned my addr 172.21.15.64:0/2983066128 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2251952374 0 0) 0x7f291009e470 con 0x7f29100a5520 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): my global_id is 174138 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28f4003520 con 0x7f29100a5520 2024-09-17T15:54:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1857360990 0 0) 0x7f2910134c90 con 0x7f2910135f70 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): my global_id is 174122 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f291009e470 con 0x7f2910135f70 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3639707490 0 0) 0x7f28f4003520 con 0x7f29100a5520 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2910134c90 con 0x7f29100a5520 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f290c001b50 con 0x7f29100a5520 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): dump: 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3386706429 0 0) 0x7f291009e470 con 0x7f2910135f70 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f28f4003520 con 0x7f2910135f70 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1159759673 0 0) 0x7f29100a07b0 con 0x7f2910135b50 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): my global_id is 174146 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f291009e470 con 0x7f2910135b50 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2908001cf0 con 0x7f2910135f70 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient(hunting): dump: 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.821 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3172465059 0 0) 0x7f2910134c90 con 0x7f29100a5520 2024-09-17T15:54:12.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 >> v1:172.21.15.64:6793/0 conn(0x7f2910135f70 legacy=0x7f29101d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 >> v1:172.21.15.100:6791/0 conn(0x7f2910135b50 legacy=0x7f2910131f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient: _finish_hunting 0 2024-09-17T15:54:12.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 monclient: found mon.d 2024-09-17T15:54:12.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:12.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:54:12.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29101347d0 con 0x7f29100a5520 2024-09-17T15:54:12.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient: _finish_auth 0 2024-09-17T15:54:12.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient: _check_auth_tickets 2024-09-17T15:54:12.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:12.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f2917e33640 5 monclient: authenticate success, global_id 174138 2024-09-17T15:54:12.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f2917e33640 10 monclient: _renew_subs 2024-09-17T15:54:12.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f2917e33640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f2917e33640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29100a8f60 con 0x7f29100a5520 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f2917e33640 10 monclient: _renew_subs 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f2917e33640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f2917e33640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f2910132670 con 0x7f29100a5520 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f290c0038c0 con 0x7f29100a5520 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.801+0000 7f29027fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.802+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f290c003e30 con 0x7f29100a5520 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.802+0000 7f29027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.802+0000 7f2903fff640 4 set_mon_vals no callback set 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.802+0000 7f29027fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.802+0000 7f29027fc640 10 monclient: dump: 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.824 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.802+0000 7f2917e33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:12.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.802+0000 7f2917e33640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f28f80065c0 con 0x7f28f8002860 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.802+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f290c004050 con 0x7f29100a5520 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f290c04eb60 con 0x7f29100a5520 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1451514695 0 0) 0x7f28f80065c0 con 0x7f28f8002860 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 10 monclient(hunting): my global_id is 154642 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2910134c90 con 0x7f28f8002860 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29163d0640 10 monclient: get_auth_request con 0x7f28f4040af0 auth_method 0 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1635746191 0 0) 0x7f2910134c90 con 0x7f28f8002860 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f28f80065c0 con 0x7f28f8002860 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2904004be0 con 0x7f28f8002860 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 10 monclient: dump: 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.827 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.828 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.828 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.828 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 795744877 0 0) 0x7f28f80065c0 con 0x7f28f8002860 2024-09-17T15:54:12.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f28f800a160 con 0x7f28f8002860 2024-09-17T15:54:12.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f29040015b0 con 0x7f28f8002860 2024-09-17T15:54:12.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:12.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:12.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 >> v1:172.21.15.64:6791/0 conn(0x7f28f8002860 legacy=0x7f28f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.803+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 >> v1:172.21.15.64:6791/0 conn(0x7f28f8002860 legacy=0x7f28f8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:12.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.809+0000 7f2917e33640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:54:12.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.809+0000 7f2917e33640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f28f8009e70 con 0x7f28f8009130 2024-09-17T15:54:12.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 446359355 0 0) 0x7f28f8009e70 con 0x7f28f8009130 2024-09-17T15:54:12.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 10 monclient(hunting): my global_id is 154651 2024-09-17T15:54:12.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28f80065c0 con 0x7f28f8009130 2024-09-17T15:54:12.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4026013646 0 0) 0x7f28f80065c0 con 0x7f28f8009130 2024-09-17T15:54:12.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f28f8009e70 con 0x7f28f8009130 2024-09-17T15:54:12.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2904005000 con 0x7f28f8009130 2024-09-17T15:54:12.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 10 monclient: dump: 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.832 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.832 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.832 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1262786116 0 0) 0x7f28f8009e70 con 0x7f28f8009130 2024-09-17T15:54:12.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 --> v1:172.21.15.64:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f28f800c410 con 0x7f28f8009130 2024-09-17T15:54:12.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 <== mon.4 v1:172.21.15.64:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 4089206612) 0x7f2904005130 con 0x7f28f8009130 2024-09-17T15:54:12.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:54:12.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:12.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 >> v1:172.21.15.64:6791/0 conn(0x7f28f8009130 legacy=0x7f28f80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.810+0000 7f29027fc640 1 -- 172.21.15.64:0/2983066128 >> v1:172.21.15.64:6791/0 conn(0x7f28f8009130 legacy=0x7f28f80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:12.833 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:54:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:54:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:54:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 46, 2024-09-17T15:54:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 144144, 2024-09-17T15:54:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:54:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:54:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:54:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:54:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:54:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:54:12.833 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:12.834 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:12.835 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:54:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:54:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:54:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:54:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:54:12.836 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:12.837 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:54:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:54:12.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:54:12.839 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:54:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:54:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:54:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:54:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:54:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:54:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.842 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:54:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:54:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.843 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:54:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:54:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:54:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:54:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.845 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:54:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:54:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.846 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:54:12.847 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:54:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:54:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:54:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.848 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.849 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:54:12.850 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:54:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:54:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:54:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:54:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 3 2024-09-17T15:54:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.853 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:54:12.853 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:12.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.811+0000 7f2917e33640 1 -- 172.21.15.64:0/2983066128 >> v1:172.21.15.100:6812/4210963484 conn(0x7f28f4040af0 legacy=0x7f28f4042fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.811+0000 7f2917e33640 10 monclient: shutdown 2024-09-17T15:54:12.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.811+0000 7f2917e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:12.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.811+0000 7f2917e33640 1 -- 172.21.15.64:0/2983066128 >> v1:172.21.15.64:6792/0 conn(0x7f29100a5520 legacy=0x7f2910131850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.811+0000 7f29163d0640 1 -- 172.21.15.64:0/2983066128 reap_dead start 2024-09-17T15:54:12.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.811+0000 7f29163d0640 1 -- 172.21.15.64:0/2983066128 reap_dead start 2024-09-17T15:54:12.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.811+0000 7f2917e33640 1 -- 172.21.15.64:0/2983066128 shutdown_connections 2024-09-17T15:54:12.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.811+0000 7f2917e33640 1 -- 172.21.15.64:0/2983066128 >> 172.21.15.64:0/2983066128 conn(0x7f291006db30 msgr2=0x7f29100a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:12.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.811+0000 7f2917e33640 1 -- 172.21.15.64:0/2983066128 shutdown_connections 2024-09-17T15:54:12.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.811+0000 7f2917e33640 1 -- 172.21.15.64:0/2983066128 wait complete. 2024-09-17T15:54:12.855 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:54:12.856 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-17T15:54:12.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-17T15:54:12.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.956+0000 7f875939b640 10 monclient: get_monmap_and_config 2024-09-17T15:54:12.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.956+0000 7f875939b640 10 monclient: build_initial_monmap 2024-09-17T15:54:12.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.956+0000 7f875939b640 10 monclient: monmap: 2024-09-17T15:54:12.958 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:12.958 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.958 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:12.956915+0000 2024-09-17T15:54:12.958 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:12.956915+0000 2024-09-17T15:54:12.958 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:12.958 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.958 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:12.958 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:12.958 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:12.958 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:12.958 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:12.958 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:12.959 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:12.959 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:12.959 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:12.959 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.956+0000 7f875939b640 1 Processor -- start 2024-09-17T15:54:12.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.956+0000 7f875939b640 1 -- start start 2024-09-17T15:54:12.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.956+0000 7f875939b640 10 monclient: init 2024-09-17T15:54:12.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:12.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 10 monclient: _add_conns ranks=[0,2,7,3,1,8,5,6,4] 2024-09-17T15:54:12.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 10 monclient(hunting): picked mon.noname-a con 0x7f87540a54b0 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:12.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 10 monclient(hunting): picked mon.noname-b con 0x7f8754135c80 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:12.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 10 monclient(hunting): picked mon.noname-i con 0x7f87541360a0 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:12.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87540aacb0 con 0x7f87540a54b0 2024-09-17T15:54:12.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f875409d780 con 0x7f8754135c80 2024-09-17T15:54:12.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f875409da60 con 0x7f87541360a0 2024-09-17T15:54:12.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:12.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 10 monclient(hunting): authenticate will time out at 1250.632236s 2024-09-17T15:54:12.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8752ffd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f87540a54b0 0x7f87540a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:34634/0 (socket says 172.21.15.64:34634) 2024-09-17T15:54:12.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8752ffd640 1 -- 172.21.15.64:0/456645097 learned_addr learned my addr 172.21.15.64:0/456645097 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:12.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f87527fc640 1 --1- 172.21.15.64:0/456645097 >> v1:172.21.15.64:6790/0 conn(0x7f8754135c80 0x7f875413df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45612/0 (socket says 172.21.15.64:45612) 2024-09-17T15:54:12.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 1 -- 172.21.15.64:0/456645097 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3183192407 0 0) 0x7f875409d780 con 0x7f8754135c80 2024-09-17T15:54:12.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient(hunting): my global_id is 125219 2024-09-17T15:54:12.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 1 -- 172.21.15.64:0/456645097 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f873c003520 con 0x7f8754135c80 2024-09-17T15:54:12.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 1 -- 172.21.15.64:0/456645097 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3125063122 0 0) 0x7f87540aacb0 con 0x7f87540a54b0 2024-09-17T15:54:12.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient(hunting): my global_id is 125109 2024-09-17T15:54:12.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 1 -- 172.21.15.64:0/456645097 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f875409d780 con 0x7f87540a54b0 2024-09-17T15:54:12.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 1 -- 172.21.15.64:0/456645097 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2364640164 0 0) 0x7f873c003520 con 0x7f8754135c80 2024-09-17T15:54:12.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 1 -- 172.21.15.64:0/456645097 >> v1:172.21.15.100:6792/0 conn(0x7f87541360a0 legacy=0x7f8754141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 1 -- 172.21.15.64:0/456645097 >> v1:172.21.15.64:6789/0 conn(0x7f87540a54b0 legacy=0x7f87540a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:54:12.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 1 monclient: found mon.noname-b 2024-09-17T15:54:12.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 1 -- 172.21.15.64:0/456645097 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f875409dd40 con 0x7f8754135c80 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient: _finish_auth 0 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 1 -- 172.21.15.64:0/456645097 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8748001830 con 0x7f8754135c80 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 5 monclient: authenticate success, global_id 125219 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient: dump: 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.970 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:12.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 1 -- 172.21.15.64:0/456645097 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8748001a50 con 0x7f8754135c80 2024-09-17T15:54:12.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:12.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f8751ffb640 1 -- 172.21.15.64:0/456645097 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8748004b20 con 0x7f8754135c80 2024-09-17T15:54:12.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.957+0000 7f875939b640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:12.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.958+0000 7f875939b640 4 set_mon_vals no callback set 2024-09-17T15:54:12.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.958+0000 7f8751ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.958+0000 7f8751ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:12.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.958+0000 7f8751ffb640 10 monclient: dump: 2024-09-17T15:54:12.972 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.972 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.972 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.972 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.972 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.958+0000 7f875939b640 10 monclient: shutdown 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.958+0000 7f875939b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.958+0000 7f875939b640 1 -- 172.21.15.64:0/456645097 >> v1:172.21.15.64:6790/0 conn(0x7f8754135c80 legacy=0x7f875413df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.958+0000 7f875939b640 1 -- 172.21.15.64:0/456645097 shutdown_connections 2024-09-17T15:54:12.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.958+0000 7f875939b640 1 -- 172.21.15.64:0/456645097 >> 172.21.15.64:0/456645097 conn(0x7f875406daa0 msgr2=0x7f87540ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:12.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.958+0000 7f875939b640 1 -- 172.21.15.64:0/456645097 shutdown_connections 2024-09-17T15:54:12.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.958+0000 7f875939b640 1 -- 172.21.15.64:0/456645097 wait complete. 2024-09-17T15:54:12.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.959+0000 7f875939b640 10 monclient: build_initial_monmap 2024-09-17T15:54:12.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.959+0000 7f875939b640 10 monclient: monmap: 2024-09-17T15:54:12.974 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:12.974 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:12.974 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:12.974 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:12.974 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:12.974 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:12.974 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:12.974 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.959+0000 7f875939b640 1 Processor -- start 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.959+0000 7f875939b640 1 -- start start 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.959+0000 7f875939b640 10 monclient: init 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.959+0000 7f875939b640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.959+0000 7f875939b640 10 monclient: _add_conns ranks=[6,0,5,8,1,4,7,2,3] 2024-09-17T15:54:12.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.959+0000 7f875939b640 10 monclient(hunting): picked mon.noname-d con 0x7f87540a54b0 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:12.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.959+0000 7f875939b640 10 monclient(hunting): picked mon.noname-a con 0x7f8754135c80 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:12.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.959+0000 7f875939b640 10 monclient(hunting): picked mon.noname-h con 0x7f87541360a0 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:12.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f875939b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87540a0880 con 0x7f8754135c80 2024-09-17T15:54:12.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f875939b640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87540a2be0 con 0x7f87541360a0 2024-09-17T15:54:12.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f875939b640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87540a0e00 con 0x7f87540a54b0 2024-09-17T15:54:12.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f875939b640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:12.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f875939b640 10 monclient(hunting): authenticate will time out at 1250.635056s 2024-09-17T15:54:12.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f87527fc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f8754135c80 0x7f87541d3010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:34640/0 (socket says 172.21.15.64:34640) 2024-09-17T15:54:12.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f87527fc640 1 -- 172.21.15.64:0/817442304 learned_addr learned my addr 172.21.15.64:0/817442304 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:12.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3603709490 0 0) 0x7f87540a0e00 con 0x7f87540a54b0 2024-09-17T15:54:12.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): my global_id is 174147 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8740003520 con 0x7f87540a54b0 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 252234984 0 0) 0x7f87540a0880 con 0x7f8754135c80 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): my global_id is 125118 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87540a0e00 con 0x7f8754135c80 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2618319557 0 0) 0x7f87540a2be0 con 0x7f87541360a0 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): my global_id is 174155 2024-09-17T15:54:12.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87540a0880 con 0x7f87541360a0 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4028904169 0 0) 0x7f8740003520 con 0x7f87540a54b0 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87540a2be0 con 0x7f87540a54b0 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8744001860 con 0x7f87540a54b0 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): dump: 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.978 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.979 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.979 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.979 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.979 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.979 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.979 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.979 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.979 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:12.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3026844595 0 0) 0x7f87540a0e00 con 0x7f8754135c80 2024-09-17T15:54:12.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8740003520 con 0x7f8754135c80 2024-09-17T15:54:12.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f87480017f0 con 0x7f8754135c80 2024-09-17T15:54:12.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): dump: 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.980 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.981 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.981 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.981 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.981 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 156634377 0 0) 0x7f87540a0880 con 0x7f87541360a0 2024-09-17T15:54:12.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87540a0e00 con 0x7f87541360a0 2024-09-17T15:54:12.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f874c0018f0 con 0x7f87541360a0 2024-09-17T15:54:12.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:12.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:54:12.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient(hunting): dump: 2024-09-17T15:54:12.981 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.981 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.981 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2482729761 0 0) 0x7f87540a2be0 con 0x7f87540a54b0 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 >> v1:172.21.15.100:6791/0 conn(0x7f87541360a0 legacy=0x7f87541d6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 >> v1:172.21.15.64:6789/0 conn(0x7f8754135c80 legacy=0x7f87541d3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 monclient: found mon.d 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87540a2010 con 0x7f87540a54b0 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f873b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f875939b640 5 monclient: authenticate success, global_id 174147 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f875939b640 10 monclient: _renew_subs 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f875939b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:54:12.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f875939b640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f87540a34c0 con 0x7f87540a54b0 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f875939b640 10 monclient: _renew_subs 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f875939b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.960+0000 7f875939b640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f875409fee0 con 0x7f87540a54b0 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.961+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8744001bc0 con 0x7f87540a54b0 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.961+0000 7f873b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.961+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8744005150 con 0x7f87540a54b0 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.961+0000 7f873b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.961+0000 7f87517fa640 4 set_mon_vals no callback set 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.961+0000 7f873b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.961+0000 7f873b7fe640 10 monclient: dump: 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.984 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.961+0000 7f875939b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:12.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.961+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f8744011a30 con 0x7f87540a54b0 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.961+0000 7f875939b640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f87340065c0 con 0x7f8734002860 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f874404eda0 con 0x7f87540a54b0 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2745324689 0 0) 0x7f87340065c0 con 0x7f8734002860 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 10 monclient(hunting): my global_id is 174156 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87540a2be0 con 0x7f8734002860 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f87537fe640 10 monclient: get_auth_request con 0x7f8740040cd0 auth_method 0 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 956955781 0 0) 0x7f87540a2be0 con 0x7f8734002860 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87340065c0 con 0x7f8734002860 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8748001170 con 0x7f8734002860 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 10 monclient: dump: 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.987 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.988 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.988 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.988 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 450478996 0 0) 0x7f87340065c0 con 0x7f8734002860 2024-09-17T15:54:12.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f873400a160 con 0x7f8734002860 2024-09-17T15:54:12.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8748007160 con 0x7f8734002860 2024-09-17T15:54:12.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:12.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:12.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 >> v1:172.21.15.64:6792/0 conn(0x7f8734002860 legacy=0x7f8734004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.962+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 >> v1:172.21.15.64:6792/0 conn(0x7f8734002860 legacy=0x7f8734004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:12.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.969+0000 7f875939b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:54:12.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.969+0000 7f875939b640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8734009e70 con 0x7f8734009130 2024-09-17T15:54:12.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2476549560 0 0) 0x7f8734009e70 con 0x7f8734009130 2024-09-17T15:54:12.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:12.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:12.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:12.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 10 monclient(hunting): my global_id is 174165 2024-09-17T15:54:12.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87340065c0 con 0x7f8734009130 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4017403835 0 0) 0x7f87340065c0 con 0x7f8734009130 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8734009e70 con 0x7f8734009130 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8748005900 con 0x7f8734009130 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 10 monclient: dump: 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:12.991 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:12.992 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:12.992 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:12.992 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:12.992 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:12.992 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:12.992 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:12.992 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:12.992 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:12.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3292745920 0 0) 0x7f8734009e70 con 0x7f8734009130 2024-09-17T15:54:12.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 --> v1:172.21.15.64:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f873400c410 con 0x7f8734009130 2024-09-17T15:54:12.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 <== mon.6 v1:172.21.15.64:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 4139617613) 0x7f874800a0c0 con 0x7f8734009130 2024-09-17T15:54:12.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:54:12.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:12.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 >> v1:172.21.15.64:6792/0 conn(0x7f8734009130 legacy=0x7f87340095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:12.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.970+0000 7f873b7fe640 1 -- 172.21.15.64:0/817442304 >> v1:172.21.15.64:6792/0 conn(0x7f8734009130 legacy=0x7f87340095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:12.993 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:54:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:54:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:54:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 46, 2024-09-17T15:54:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 15912, 2024-09-17T15:54:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:54:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 0, 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:54:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:54:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:54:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:54:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:54:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:54:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:54:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:54:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:54:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:54:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:54:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:54:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:54:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:54:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:54:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:54:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:54:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:54:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:54:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:54:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:54:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:54:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:54:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:54:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:54:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:54:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:54:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:54:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:54:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:54:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:54:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:54:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:54:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:54:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:54:13.013 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:13.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.971+0000 7f875939b640 1 -- 172.21.15.64:0/817442304 >> v1:172.21.15.100:6812/4210963484 conn(0x7f8740040cd0 legacy=0x7f8740043190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.971+0000 7f875939b640 10 monclient: shutdown 2024-09-17T15:54:13.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.971+0000 7f875939b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:13.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.971+0000 7f875939b640 1 -- 172.21.15.64:0/817442304 >> v1:172.21.15.64:6792/0 conn(0x7f87540a54b0 legacy=0x7f87541ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.971+0000 7f87537fe640 1 -- 172.21.15.64:0/817442304 reap_dead start 2024-09-17T15:54:13.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.971+0000 7f87537fe640 1 -- 172.21.15.64:0/817442304 reap_dead start 2024-09-17T15:54:13.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.971+0000 7f875939b640 1 -- 172.21.15.64:0/817442304 shutdown_connections 2024-09-17T15:54:13.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.971+0000 7f875939b640 1 -- 172.21.15.64:0/817442304 >> 172.21.15.64:0/817442304 conn(0x7f875406daa0 msgr2=0x7f87540a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:13.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.971+0000 7f875939b640 1 -- 172.21.15.64:0/817442304 shutdown_connections 2024-09-17T15:54:13.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:12.971+0000 7f875939b640 1 -- 172.21.15.64:0/817442304 wait complete. 2024-09-17T15:54:13.015 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:54:13.015 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-17T15:54:13.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-17T15:54:13.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.116+0000 7f8f06532640 10 monclient: get_monmap_and_config 2024-09-17T15:54:13.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.116+0000 7f8f06532640 10 monclient: build_initial_monmap 2024-09-17T15:54:13.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.116+0000 7f8f06532640 10 monclient: monmap: 2024-09-17T15:54:13.119 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:13.119 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.119 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:13.117706+0000 2024-09-17T15:54:13.119 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:13.117706+0000 2024-09-17T15:54:13.119 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:13.119 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.119 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:13.119 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:13.119 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:13.119 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:13.119 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:13.119 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:13.119 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:13.120 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:13.120 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:13.120 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.117+0000 7f8f06532640 1 Processor -- start 2024-09-17T15:54:13.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.117+0000 7f8f06532640 1 -- start start 2024-09-17T15:54:13.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.117+0000 7f8f06532640 10 monclient: init 2024-09-17T15:54:13.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8f06532640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:13.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8f06532640 10 monclient: _add_conns ranks=[3,5,1,7,2,4,6,0,8] 2024-09-17T15:54:13.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8f06532640 10 monclient(hunting): picked mon.noname-g con 0x7f8f000a5520 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:13.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8f06532640 10 monclient(hunting): picked mon.noname-h con 0x7f8f00135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:13.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8f06532640 10 monclient(hunting): picked mon.noname-f con 0x7f8f00135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:13.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8f06532640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f0009d6f0 con 0x7f8f00135f70 2024-09-17T15:54:13.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8f06532640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f0009d9d0 con 0x7f8f000a5520 2024-09-17T15:54:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8f06532640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f0009dcb0 con 0x7f8f00135b50 2024-09-17T15:54:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8f06532640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8f06532640 10 monclient(hunting): authenticate will time out at 1250.793193s 2024-09-17T15:54:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efffff640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f8f000a5520 0x7f8f000a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:46118/0 (socket says 172.21.15.64:46118) 2024-09-17T15:54:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efffff640 1 -- 172.21.15.64:0/746949479 learned_addr learned my addr 172.21.15.64:0/746949479 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 1 -- 172.21.15.64:0/746949479 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3922053970 0 0) 0x7f8f0009d9d0 con 0x7f8f000a5520 2024-09-17T15:54:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient(hunting): my global_id is 115905 2024-09-17T15:54:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 1 -- 172.21.15.64:0/746949479 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ee8003520 con 0x7f8f000a5520 2024-09-17T15:54:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 1 -- 172.21.15.64:0/746949479 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2616481084 0 0) 0x7f8f0009dcb0 con 0x7f8f00135b50 2024-09-17T15:54:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient(hunting): my global_id is 174164 2024-09-17T15:54:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 1 -- 172.21.15.64:0/746949479 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f0009d9d0 con 0x7f8f00135b50 2024-09-17T15:54:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 1 -- 172.21.15.64:0/746949479 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3282925780 0 0) 0x7f8f0009d6f0 con 0x7f8f00135f70 2024-09-17T15:54:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient(hunting): my global_id is 6913 2024-09-17T15:54:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 1 -- 172.21.15.64:0/746949479 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f0009dcb0 con 0x7f8f00135f70 2024-09-17T15:54:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 1 -- 172.21.15.64:0/746949479 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2121908601 0 0) 0x7f8f0009d9d0 con 0x7f8f00135b50 2024-09-17T15:54:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 1 -- 172.21.15.64:0/746949479 >> v1:172.21.15.100:6790/0 conn(0x7f8f000a5520 legacy=0x7f8f000a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 1 -- 172.21.15.64:0/746949479 >> v1:172.21.15.100:6789/0 conn(0x7f8f00135f70 legacy=0x7f8f001415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient: _finish_hunting 0 2024-09-17T15:54:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 1 monclient: found mon.noname-h 2024-09-17T15:54:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.100:6791/0 2024-09-17T15:54:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 1 -- 172.21.15.64:0/746949479 --> v1:172.21.15.100:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0009ccd0 con 0x7f8f00135b50 2024-09-17T15:54:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.118+0000 7f8efeffd640 10 monclient: _finish_auth 0 2024-09-17T15:54:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8efeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:54:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8efeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8efeffd640 1 -- 172.21.15.64:0/746949479 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8ef4001810 con 0x7f8f00135b50 2024-09-17T15:54:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8f06532640 5 monclient: authenticate success, global_id 174164 2024-09-17T15:54:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8f06532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8efeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8efeffd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8efeffd640 10 monclient: dump: 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.132 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.133 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8f06532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8efeffd640 1 -- 172.21.15.64:0/746949479 <== mon.5 v1:172.21.15.100:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ef40019b0 con 0x7f8f00135b50 2024-09-17T15:54:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8efeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8efeffd640 1 -- 172.21.15.64:0/746949479 <== mon.5 v1:172.21.15.100:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8ef4003950 con 0x7f8f00135b50 2024-09-17T15:54:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8efeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8efeffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:54:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8efeffd640 10 monclient: dump: 2024-09-17T15:54:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8f06532640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8f06532640 4 set_mon_vals no callback set 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8f06532640 10 monclient: shutdown 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8f06532640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8f06532640 1 -- 172.21.15.64:0/746949479 >> v1:172.21.15.100:6791/0 conn(0x7f8f00135b50 legacy=0x7f8f0013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8f06532640 1 -- 172.21.15.64:0/746949479 shutdown_connections 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8f06532640 1 -- 172.21.15.64:0/746949479 >> 172.21.15.64:0/746949479 conn(0x7f8f0006db30 msgr2=0x7f8f000ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8f06532640 1 -- 172.21.15.64:0/746949479 shutdown_connections 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.119+0000 7f8f06532640 1 -- 172.21.15.64:0/746949479 wait complete. 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.120+0000 7f8f06532640 10 monclient: build_initial_monmap 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.120+0000 7f8f06532640 10 monclient: monmap: 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:13.136 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:13.136 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:13.136 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:13.136 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:13.136 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:13.136 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:13.136 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:13.136 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:13.137 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:13.137 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:13.137 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:13.137 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.120+0000 7f8f06532640 1 Processor -- start 2024-09-17T15:54:13.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.120+0000 7f8f06532640 1 -- start start 2024-09-17T15:54:13.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.120+0000 7f8f06532640 10 monclient: init 2024-09-17T15:54:13.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8f06532640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:13.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8f06532640 10 monclient: _add_conns ranks=[6,5,3,7,8,1,2,4,0] 2024-09-17T15:54:13.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8f06532640 10 monclient(hunting): picked mon.noname-d con 0x7f8f000a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:13.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8f06532640 10 monclient(hunting): picked mon.noname-h con 0x7f8f00135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:13.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8f06532640 10 monclient(hunting): picked mon.noname-g con 0x7f8f00135f70 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:13.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8f06532640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f001cb050 con 0x7f8f00135f70 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8f06532640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f00135210 con 0x7f8f00135b50 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8f06532640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f00134e70 con 0x7f8f000a5520 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8f06532640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8f06532640 10 monclient(hunting): authenticate will time out at 1250.796340s 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efffff640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f8f000a5520 0x7f8f001ca230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:37228/0 (socket says 172.21.15.64:37228) 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efffff640 1 -- 172.21.15.64:0/353760781 learned_addr learned my addr 172.21.15.64:0/353760781 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 473722183 0 0) 0x7f8f00134e70 con 0x7f8f000a5520 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): my global_id is 174174 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8eec003550 con 0x7f8f000a5520 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2826908601 0 0) 0x7f8f001cb050 con 0x7f8f00135f70 2024-09-17T15:54:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): my global_id is 115914 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f00134e70 con 0x7f8f00135f70 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2585219985 0 0) 0x7f8f00135210 con 0x7f8f00135b50 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): my global_id is 174173 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f001cb050 con 0x7f8f00135b50 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4238955914 0 0) 0x7f8eec003550 con 0x7f8f000a5520 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f00135210 con 0x7f8f000a5520 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8ef0001610 con 0x7f8f000a5520 2024-09-17T15:54:13.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): dump: 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.140 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.141 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.121+0000 7f8efcff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1980722010 0 0) 0x7f8f001cb050 con 0x7f8f00135b50 2024-09-17T15:54:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8eec003550 con 0x7f8f00135b50 2024-09-17T15:54:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8ef4001410 con 0x7f8f00135b50 2024-09-17T15:54:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:54:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 10 monclient(hunting): dump: 2024-09-17T15:54:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.142 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 341563533 0 0) 0x7f8f00135210 con 0x7f8f000a5520 2024-09-17T15:54:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 >> v1:172.21.15.100:6791/0 conn(0x7f8f00135b50 legacy=0x7f8f001ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 >> v1:172.21.15.100:6790/0 conn(0x7f8f00135f70 legacy=0x7f8f001d5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 10 monclient: _finish_hunting 0 2024-09-17T15:54:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 1 monclient: found mon.d 2024-09-17T15:54:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:54:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0009f200 con 0x7f8f000a5520 2024-09-17T15:54:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 10 monclient: _finish_auth 0 2024-09-17T15:54:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 10 monclient: _check_auth_tickets 2024-09-17T15:54:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8f06532640 5 monclient: authenticate success, global_id 174174 2024-09-17T15:54:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8f06532640 10 monclient: _renew_subs 2024-09-17T15:54:13.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8f06532640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:54:13.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8f06532640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0009fdb0 con 0x7f8f000a5520 2024-09-17T15:54:13.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8f06532640 10 monclient: _renew_subs 2024-09-17T15:54:13.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8f06532640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:54:13.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8f06532640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f8f0009e470 con 0x7f8f000a5520 2024-09-17T15:54:13.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ef0004a90 con 0x7f8f000a5520 2024-09-17T15:54:13.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:13.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8ef0004f80 con 0x7f8f000a5520 2024-09-17T15:54:13.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:13.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efe7fc640 4 set_mon_vals no callback set 2024-09-17T15:54:13.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.122+0000 7f8efcff9640 10 monclient: dump: 2024-09-17T15:54:13.144 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.145 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8f06532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8f06532640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8ee00065c0 con 0x7f8ee0002860 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f8ef0011860 con 0x7f8f000a5520 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f8ef004ea90 con 0x7f8f000a5520 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2360693177 0 0) 0x7f8ee00065c0 con 0x7f8ee0002860 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 10 monclient(hunting): my global_id is 174131 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f00135210 con 0x7f8ee0002860 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8f04acf640 10 monclient: get_auth_request con 0x7f8eec040840 auth_method 0 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3797542693 0 0) 0x7f8f00135210 con 0x7f8ee0002860 2024-09-17T15:54:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ee00065c0 con 0x7f8ee0002860 2024-09-17T15:54:13.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8ef4001860 con 0x7f8ee0002860 2024-09-17T15:54:13.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:13.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.123+0000 7f8efcff9640 10 monclient: dump: 2024-09-17T15:54:13.147 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.147 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.147 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.147 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.147 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.147 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.148 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.148 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.148 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.148 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.148 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.148 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.148 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.148 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.148 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.148 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.124+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3433354345 0 0) 0x7f8ee00065c0 con 0x7f8ee0002860 2024-09-17T15:54:13.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.124+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.64:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ee000a160 con 0x7f8ee0002860 2024-09-17T15:54:13.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.124+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8ef4006480 con 0x7f8ee0002860 2024-09-17T15:54:13.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.124+0000 7f8efcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:13.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.124+0000 7f8efcff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:13.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.124+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 >> v1:172.21.15.64:6793/0 conn(0x7f8ee0002860 legacy=0x7f8ee0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.124+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 >> v1:172.21.15.64:6793/0 conn(0x7f8ee0002860 legacy=0x7f8ee0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:13.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8f06532640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:54:13.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8f06532640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8ee0009e70 con 0x7f8ee0009130 2024-09-17T15:54:13.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 15490167 0 0) 0x7f8ee0009e70 con 0x7f8ee0009130 2024-09-17T15:54:13.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8efcff9640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8efcff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8efcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8efcff9640 10 monclient(hunting): my global_id is 174140 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ee00065c0 con 0x7f8ee0009130 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2384398335 0 0) 0x7f8ee00065c0 con 0x7f8ee0009130 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ee0009e70 con 0x7f8ee0009130 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8ef4004670 con 0x7f8ee0009130 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8efcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8efcff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.130+0000 7f8efcff9640 10 monclient: dump: 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.151 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.152 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.152 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.152 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.152 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.152 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.152 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.152 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.152 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.152 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.152 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.152 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.152 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3520970982 0 0) 0x7f8ee0009e70 con 0x7f8ee0009130 2024-09-17T15:54:13.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 --> v1:172.21.15.64:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8ee000c410 con 0x7f8ee0009130 2024-09-17T15:54:13.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 <== mon.8 v1:172.21.15.64:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 1588367150) 0x7f8ef4006630 con 0x7f8ee0009130 2024-09-17T15:54:13.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8efcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:54:13.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8efcff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:13.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 >> v1:172.21.15.64:6793/0 conn(0x7f8ee0009130 legacy=0x7f8ee00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8efcff9640 1 -- 172.21.15.64:0/353760781 >> v1:172.21.15.64:6793/0 conn(0x7f8ee0009130 legacy=0x7f8ee00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:13.153 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:54:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:54:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 46, 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 16073, 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:54:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:54:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:54:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:54:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:54:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:54:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:54:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:54:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:54:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:54:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:54:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:54:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:54:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:54:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:54:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:54:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:54:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:54:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:54:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:54:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:54:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:54:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:54:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:54:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:54:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:54:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:54:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:54:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:54:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:54:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:54:13.172 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:13.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8f06532640 1 -- 172.21.15.64:0/353760781 >> v1:172.21.15.100:6812/4210963484 conn(0x7f8eec040840 legacy=0x7f8eec042d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8f06532640 10 monclient: shutdown 2024-09-17T15:54:13.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8f06532640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:13.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8f06532640 1 -- 172.21.15.64:0/353760781 >> v1:172.21.15.64:6792/0 conn(0x7f8f000a5520 legacy=0x7f8f001ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8f04acf640 1 -- 172.21.15.64:0/353760781 reap_dead start 2024-09-17T15:54:13.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8f04acf640 1 -- 172.21.15.64:0/353760781 reap_dead start 2024-09-17T15:54:13.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8f06532640 1 -- 172.21.15.64:0/353760781 shutdown_connections 2024-09-17T15:54:13.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8f06532640 1 -- 172.21.15.64:0/353760781 >> 172.21.15.64:0/353760781 conn(0x7f8f0006db30 msgr2=0x7f8f001334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:13.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8f06532640 1 -- 172.21.15.64:0/353760781 shutdown_connections 2024-09-17T15:54:13.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.131+0000 7f8f06532640 1 -- 172.21.15.64:0/353760781 wait complete. 2024-09-17T15:54:13.175 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:54:13.175 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-17T15:54:13.175 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-17T15:54:13.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.275+0000 7f8561d10640 10 monclient: get_monmap_and_config 2024-09-17T15:54:13.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.275+0000 7f8561d10640 10 monclient: build_initial_monmap 2024-09-17T15:54:13.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.275+0000 7f8561d10640 10 monclient: monmap: 2024-09-17T15:54:13.277 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:13.277 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.277 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:13.275954+0000 2024-09-17T15:54:13.277 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:13.275954+0000 2024-09-17T15:54:13.277 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:13.277 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.277 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:13.277 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:13.277 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:13.277 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:13.277 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:13.278 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:13.278 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:13.278 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:13.278 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:13.278 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.275+0000 7f8561d10640 1 Processor -- start 2024-09-17T15:54:13.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.275+0000 7f8561d10640 1 -- start start 2024-09-17T15:54:13.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.275+0000 7f8561d10640 10 monclient: init 2024-09-17T15:54:13.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f8561d10640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:13.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f8561d10640 10 monclient: _add_conns ranks=[0,5,2,4,7,8,6,1,3] 2024-09-17T15:54:13.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f8561d10640 10 monclient(hunting): picked mon.noname-a con 0x7f855c0a54b0 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:13.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f8561d10640 10 monclient(hunting): picked mon.noname-h con 0x7f855c135c80 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:13.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f8561d10640 10 monclient(hunting): picked mon.noname-b con 0x7f855c1360a0 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:13.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f8561d10640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855c0aacb0 con 0x7f855c0a54b0 2024-09-17T15:54:13.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f8561d10640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855c09d780 con 0x7f855c1360a0 2024-09-17T15:54:13.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f8561d10640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855c09da60 con 0x7f855c135c80 2024-09-17T15:54:13.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f8561d10640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:13.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f8561d10640 10 monclient(hunting): authenticate will time out at 1250.951564s 2024-09-17T15:54:13.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855b7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f855c0a54b0 0x7f855c0a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:34654/0 (socket says 172.21.15.64:34654) 2024-09-17T15:54:13.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855b7fe640 1 -- 172.21.15.64:0/2813213142 learned_addr learned my addr 172.21.15.64:0/2813213142 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:13.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855a7fc640 1 -- 172.21.15.64:0/2813213142 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2505916571 0 0) 0x7f855c0aacb0 con 0x7f855c0a54b0 2024-09-17T15:54:13.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855a7fc640 10 monclient(hunting): my global_id is 125127 2024-09-17T15:54:13.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855a7fc640 1 -- 172.21.15.64:0/2813213142 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8548003520 con 0x7f855c0a54b0 2024-09-17T15:54:13.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855a7fc640 1 -- 172.21.15.64:0/2813213142 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2414135817 0 0) 0x7f855c09d780 con 0x7f855c1360a0 2024-09-17T15:54:13.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855a7fc640 10 monclient(hunting): my global_id is 125228 2024-09-17T15:54:13.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.276+0000 7f855a7fc640 1 -- 172.21.15.64:0/2813213142 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f855c0aacb0 con 0x7f855c1360a0 2024-09-17T15:54:13.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 1 -- 172.21.15.64:0/2813213142 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4024443101 0 0) 0x7f855c09da60 con 0x7f855c135c80 2024-09-17T15:54:13.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient(hunting): my global_id is 174182 2024-09-17T15:54:13.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 1 -- 172.21.15.64:0/2813213142 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f855c09d780 con 0x7f855c135c80 2024-09-17T15:54:13.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 1 -- 172.21.15.64:0/2813213142 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3298729505 0 0) 0x7f8548003520 con 0x7f855c0a54b0 2024-09-17T15:54:13.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 1 -- 172.21.15.64:0/2813213142 >> v1:172.21.15.100:6791/0 conn(0x7f855c135c80 legacy=0x7f855c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 1 -- 172.21.15.64:0/2813213142 >> v1:172.21.15.64:6790/0 conn(0x7f855c1360a0 legacy=0x7f855c141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:54:13.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 1 monclient: found mon.noname-a 2024-09-17T15:54:13.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:13.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:54:13.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 1 -- 172.21.15.64:0/2813213142 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855c09dd40 con 0x7f855c0a54b0 2024-09-17T15:54:13.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:54:13.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:54:13.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:13.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f8561d10640 5 monclient: authenticate success, global_id 125127 2024-09-17T15:54:13.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 1 -- 172.21.15.64:0/2813213142 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8554001b30 con 0x7f855c0a54b0 2024-09-17T15:54:13.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:54:13.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient: dump: 2024-09-17T15:54:13.293 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.293 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.293 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.293 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.293 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.293 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.293 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.293 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f8561d10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 1 -- 172.21.15.64:0/2813213142 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8554001cd0 con 0x7f855c0a54b0 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 1 -- 172.21.15.64:0/2813213142 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8554004f60 con 0x7f855c0a54b0 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f8561d10640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f8561d10640 4 set_mon_vals no callback set 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f855a7fc640 10 monclient: dump: 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.295 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.296 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.296 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.296 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.296 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.296 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f8561d10640 10 monclient: shutdown 2024-09-17T15:54:13.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f8561d10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:13.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f8561d10640 1 -- 172.21.15.64:0/2813213142 >> v1:172.21.15.64:6789/0 conn(0x7f855c0a54b0 legacy=0x7f855c0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f8561d10640 1 -- 172.21.15.64:0/2813213142 shutdown_connections 2024-09-17T15:54:13.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f8561d10640 1 -- 172.21.15.64:0/2813213142 >> 172.21.15.64:0/2813213142 conn(0x7f855c06daa0 msgr2=0x7f855c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:13.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f8561d10640 1 -- 172.21.15.64:0/2813213142 shutdown_connections 2024-09-17T15:54:13.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.277+0000 7f8561d10640 1 -- 172.21.15.64:0/2813213142 wait complete. 2024-09-17T15:54:13.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.278+0000 7f8561d10640 10 monclient: build_initial_monmap 2024-09-17T15:54:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.278+0000 7f8561d10640 10 monclient: monmap: 2024-09-17T15:54:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.278+0000 7f8561d10640 1 Processor -- start 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8561d10640 1 -- start start 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8561d10640 10 monclient: init 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8561d10640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8561d10640 10 monclient: _add_conns ranks=[4,2,5,3,7,6,1,0,8] 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8561d10640 10 monclient(hunting): picked mon.noname-c con 0x7f855c0a54b0 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8561d10640 10 monclient(hunting): picked mon.noname-b con 0x7f855c135c80 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8561d10640 10 monclient(hunting): picked mon.noname-h con 0x7f855c1360a0 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8561d10640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855c09f0a0 con 0x7f855c135c80 2024-09-17T15:54:13.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8561d10640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855c0a6ec0 con 0x7f855c0a54b0 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8561d10640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855c0a6900 con 0x7f855c1360a0 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8561d10640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8561d10640 10 monclient(hunting): authenticate will time out at 1250.954597s 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f855affd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f855c135c80 0x7f855c132210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45628/0 (socket says 172.21.15.64:45628) 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f855affd640 1 -- 172.21.15.64:0/2818939430 learned_addr learned my addr 172.21.15.64:0/2818939430 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1353399331 0 0) 0x7f855c09dd40 con 0x7f855c0a54b0 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 10 monclient(hunting): my global_id is 154660 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f853c003520 con 0x7f855c0a54b0 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4267927138 0 0) 0x7f855c09d780 con 0x7f855c135c80 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 10 monclient(hunting): my global_id is 125237 2024-09-17T15:54:13.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f855c09dd40 con 0x7f855c135c80 2024-09-17T15:54:13.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775062990 0 0) 0x7f854c001210 con 0x7f855c1360a0 2024-09-17T15:54:13.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 10 monclient(hunting): my global_id is 174191 2024-09-17T15:54:13.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.279+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f855c09d780 con 0x7f855c1360a0 2024-09-17T15:54:13.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3862595958 0 0) 0x7f853c003520 con 0x7f855c0a54b0 2024-09-17T15:54:13.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f854c001210 con 0x7f855c0a54b0 2024-09-17T15:54:13.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8554001a70 con 0x7f855c0a54b0 2024-09-17T15:54:13.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient(hunting): dump: 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.301 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3987658676 0 0) 0x7f855c09dd40 con 0x7f855c135c80 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f853c003520 con 0x7f855c135c80 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8544001b10 con 0x7f855c135c80 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient(hunting): dump: 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.303 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.303 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.303 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.303 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.303 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.303 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.303 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.303 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.303 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.303 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.303 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.303 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.303 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 847780828 0 0) 0x7f855c09d780 con 0x7f855c1360a0 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f855c09dd40 con 0x7f855c1360a0 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f854c001bc0 con 0x7f855c1360a0 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient(hunting): dump: 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.304 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.305 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.305 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.305 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.305 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.305 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.305 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.305 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.305 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3363676066 0 0) 0x7f854c001210 con 0x7f855c0a54b0 2024-09-17T15:54:13.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 >> v1:172.21.15.100:6791/0 conn(0x7f855c1360a0 legacy=0x7f855c132920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 >> v1:172.21.15.64:6790/0 conn(0x7f855c135c80 legacy=0x7f855c132210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient: _finish_hunting 0 2024-09-17T15:54:13.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 monclient: found mon.c 2024-09-17T15:54:13.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:13.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:13.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855c0a0520 con 0x7f855c0a54b0 2024-09-17T15:54:13.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient: _finish_auth 0 2024-09-17T15:54:13.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient: _check_auth_tickets 2024-09-17T15:54:13.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:13.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8561d10640 5 monclient: authenticate success, global_id 154660 2024-09-17T15:54:13.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8561d10640 10 monclient: _renew_subs 2024-09-17T15:54:13.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8561d10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:13.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8561d10640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f855c09e4a0 con 0x7f855c0a54b0 2024-09-17T15:54:13.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8561d10640 10 monclient: _renew_subs 2024-09-17T15:54:13.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8561d10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:13.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8561d10640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f855c09f2d0 con 0x7f855c0a54b0 2024-09-17T15:54:13.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8554004e90 con 0x7f855c0a54b0 2024-09-17T15:54:13.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:13.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8554001cd0 con 0x7f855c0a54b0 2024-09-17T15:54:13.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:13.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8559ffb640 4 set_mon_vals no callback set 2024-09-17T15:54:13.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.280+0000 7f8553fff640 10 monclient: dump: 2024-09-17T15:54:13.307 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.307 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.307 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.307 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.307 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.307 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.281+0000 7f8561d10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.281+0000 7f8561d10640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f85400065c0 con 0x7f8540002860 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.281+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f8554011ca0 con 0x7f855c0a54b0 2024-09-17T15:54:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.281+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f855404ebd0 con 0x7f855c0a54b0 2024-09-17T15:54:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.281+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1904865241 0 0) 0x7f85400065c0 con 0x7f8540002860 2024-09-17T15:54:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.281+0000 7f8553fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.281+0000 7f8553fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.281+0000 7f8553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.281+0000 7f8553fff640 10 monclient(hunting): my global_id is 6922 2024-09-17T15:54:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.281+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f854c001210 con 0x7f8540002860 2024-09-17T15:54:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.281+0000 7f855bfff640 10 monclient: get_auth_request con 0x7f853c040de0 auth_method 0 2024-09-17T15:54:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.282+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1876543443 0 0) 0x7f854c001210 con 0x7f8540002860 2024-09-17T15:54:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.282+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85400065c0 con 0x7f8540002860 2024-09-17T15:54:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.282+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8544003eb0 con 0x7f8540002860 2024-09-17T15:54:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.282+0000 7f8553fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.282+0000 7f8553fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:54:13.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.282+0000 7f8553fff640 10 monclient: dump: 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.311 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.311 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.282+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 368442034 0 0) 0x7f85400065c0 con 0x7f8540002860 2024-09-17T15:54:13.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.282+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.100:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f854000a160 con 0x7f8540002860 2024-09-17T15:54:13.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.282+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8544004160 con 0x7f8540002860 2024-09-17T15:54:13.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.282+0000 7f8553fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:13.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.282+0000 7f8553fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:13.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.282+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 >> v1:172.21.15.100:6789/0 conn(0x7f8540002860 legacy=0x7f8540004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.282+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 >> v1:172.21.15.100:6789/0 conn(0x7f8540002860 legacy=0x7f8540004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:13.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.288+0000 7f8561d10640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:54:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.288+0000 7f8561d10640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8540009e00 con 0x7f85400090e0 2024-09-17T15:54:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2384823538 0 0) 0x7f8540009e00 con 0x7f85400090e0 2024-09-17T15:54:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 10 monclient(hunting): my global_id is 6931 2024-09-17T15:54:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85400065c0 con 0x7f85400090e0 2024-09-17T15:54:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1850539567 0 0) 0x7f85400065c0 con 0x7f85400090e0 2024-09-17T15:54:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8540009e00 con 0x7f85400090e0 2024-09-17T15:54:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8544005b00 con 0x7f85400090e0 2024-09-17T15:54:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:54:13.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 10 monclient: dump: 2024-09-17T15:54:13.314 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.314 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.314 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.314 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.314 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.314 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.314 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.314 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.314 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.314 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.314 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.314 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.314 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.315 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.315 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.315 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.1 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1651362871 0 0) 0x7f8540009e00 con 0x7f85400090e0 2024-09-17T15:54:13.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.289+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 --> v1:172.21.15.100:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f854000c3a0 con 0x7f85400090e0 2024-09-17T15:54:13.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 <== mon.1 v1:172.21.15.100:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7713 (unknown 0 0 3973592641) 0x7f8544007e30 con 0x7f85400090e0 2024-09-17T15:54:13.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f8553fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:54:13.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f8553fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:13.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 >> v1:172.21.15.100:6789/0 conn(0x7f85400090e0 legacy=0x7f8540009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f8553fff640 1 -- 172.21.15.64:0/2818939430 >> v1:172.21.15.100:6789/0 conn(0x7f85400090e0 legacy=0x7f8540009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:13.315 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:54:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:54:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "leader", 2024-09-17T15:54:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 46, 2024-09-17T15:54:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 509941, 2024-09-17T15:54:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:54:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:54:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:54:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:54:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:54:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:54:13.316 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:54:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:54:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:54:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:54:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:54:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:54:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:54:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:54:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:54:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.320 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:54:13.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:54:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:54:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:54:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:54:13.322 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:54:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:54:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:54:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:54:13.326 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:54:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:54:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:54:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:54:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:54:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:54:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:54:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.330 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:54:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:54:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:54:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:54:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.331 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:54:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:54:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:54:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:54:13.333 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.334 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:54:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:54:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.335 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.336 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:54:13.336 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:13.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f8561d10640 1 -- 172.21.15.64:0/2818939430 >> v1:172.21.15.100:6812/4210963484 conn(0x7f853c040de0 legacy=0x7f853c0432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f8561d10640 10 monclient: shutdown 2024-09-17T15:54:13.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f8561d10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:13.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f8561d10640 1 -- 172.21.15.64:0/2818939430 >> v1:172.21.15.64:6791/0 conn(0x7f855c0a54b0 legacy=0x7f855c131b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f855bfff640 1 -- 172.21.15.64:0/2818939430 reap_dead start 2024-09-17T15:54:13.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f855bfff640 1 -- 172.21.15.64:0/2818939430 reap_dead start 2024-09-17T15:54:13.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f8561d10640 1 -- 172.21.15.64:0/2818939430 shutdown_connections 2024-09-17T15:54:13.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f8561d10640 1 -- 172.21.15.64:0/2818939430 >> 172.21.15.64:0/2818939430 conn(0x7f855c06daa0 msgr2=0x7f855c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:13.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.290+0000 7f8561d10640 1 -- 172.21.15.64:0/2818939430 shutdown_connections 2024-09-17T15:54:13.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.291+0000 7f8561d10640 1 -- 172.21.15.64:0/2818939430 wait complete. 2024-09-17T15:54:13.339 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:54:13.339 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-17T15:54:13.339 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-17T15:54:13.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.439+0000 7f0fff04f640 10 monclient: get_monmap_and_config 2024-09-17T15:54:13.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.439+0000 7f0fff04f640 10 monclient: build_initial_monmap 2024-09-17T15:54:13.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.439+0000 7f0fff04f640 10 monclient: monmap: 2024-09-17T15:54:13.441 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:13.441 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.441 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:13.440380+0000 2024-09-17T15:54:13.441 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:13.440380+0000 2024-09-17T15:54:13.442 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:13.442 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.442 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:13.442 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:13.442 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:13.442 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:13.442 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:13.442 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:13.442 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:13.442 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:13.442 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:13.442 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.440+0000 7f0fff04f640 1 Processor -- start 2024-09-17T15:54:13.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.440+0000 7f0fff04f640 1 -- start start 2024-09-17T15:54:13.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.440+0000 7f0fff04f640 10 monclient: init 2024-09-17T15:54:13.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.440+0000 7f0fff04f640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:13.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.440+0000 7f0fff04f640 10 monclient: _add_conns ranks=[7,5,2,0,4,1,3,6,8] 2024-09-17T15:54:13.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.440+0000 7f0fff04f640 10 monclient(hunting): picked mon.noname-i con 0x7f0ff80a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:13.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.440+0000 7f0fff04f640 10 monclient(hunting): picked mon.noname-h con 0x7f0ff8135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:13.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.440+0000 7f0fff04f640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff8135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:13.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.440+0000 7f0fff04f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ff809d6f0 con 0x7f0ff8135f70 2024-09-17T15:54:13.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.440+0000 7f0fff04f640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ff809d9d0 con 0x7f0ff8135b50 2024-09-17T15:54:13.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.440+0000 7f0fff04f640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ff809dcb0 con 0x7f0ff80a5520 2024-09-17T15:54:13.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.440+0000 7f0fff04f640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:13.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.440+0000 7f0fff04f640 10 monclient(hunting): authenticate will time out at 1251.115896s 2024-09-17T15:54:13.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ffd5ec640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f0ff8135f70 0x7f0ff81415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45640/0 (socket says 172.21.15.64:45640) 2024-09-17T15:54:13.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ffd5ec640 1 -- 172.21.15.64:0/2887060434 learned_addr learned my addr 172.21.15.64:0/2887060434 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:13.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 -- 172.21.15.64:0/2887060434 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 466458916 0 0) 0x7f0ff809d6f0 con 0x7f0ff8135f70 2024-09-17T15:54:13.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient(hunting): my global_id is 125246 2024-09-17T15:54:13.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 -- 172.21.15.64:0/2887060434 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fe8003520 con 0x7f0ff8135f70 2024-09-17T15:54:13.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 -- 172.21.15.64:0/2887060434 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4039070935 0 0) 0x7f0ff809d9d0 con 0x7f0ff8135b50 2024-09-17T15:54:13.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient(hunting): my global_id is 174200 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 -- 172.21.15.64:0/2887060434 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ff809d6f0 con 0x7f0ff8135b50 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 -- 172.21.15.64:0/2887060434 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1356093608 0 0) 0x7f0ff809dcb0 con 0x7f0ff80a5520 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient(hunting): my global_id is 174139 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 -- 172.21.15.64:0/2887060434 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ff809d9d0 con 0x7f0ff80a5520 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 -- 172.21.15.64:0/2887060434 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3956736273 0 0) 0x7f0fe8003520 con 0x7f0ff8135f70 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 -- 172.21.15.64:0/2887060434 >> v1:172.21.15.100:6792/0 conn(0x7f0ff80a5520 legacy=0x7f0ff80a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 -- 172.21.15.64:0/2887060434 >> v1:172.21.15.100:6791/0 conn(0x7f0ff8135b50 legacy=0x7f0ff813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:54:13.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 monclient: found mon.noname-b 2024-09-17T15:54:13.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:13.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:54:13.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 -- 172.21.15.64:0/2887060434 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff809ccd0 con 0x7f0ff8135f70 2024-09-17T15:54:13.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient: _finish_auth 0 2024-09-17T15:54:13.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:54:13.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:13.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 -- 172.21.15.64:0/2887060434 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0ff0001bb0 con 0x7f0ff8135f70 2024-09-17T15:54:13.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0fff04f640 5 monclient: authenticate success, global_id 125246 2024-09-17T15:54:13.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0fff04f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:13.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient: dump: 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.455 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.456 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.456 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.456 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0fff04f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:13.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 -- 172.21.15.64:0/2887060434 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ff0001dd0 con 0x7f0ff8135f70 2024-09-17T15:54:13.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:13.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0fff04f640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:13.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0fff04f640 4 set_mon_vals no callback set 2024-09-17T15:54:13.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 1 -- 172.21.15.64:0/2887060434 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0ff0004f60 con 0x7f0ff8135f70 2024-09-17T15:54:13.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:13.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.441+0000 7f0ff77fe640 10 monclient: dump: 2024-09-17T15:54:13.456 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.457 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.442+0000 7f0fff04f640 10 monclient: shutdown 2024-09-17T15:54:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.442+0000 7f0fff04f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.442+0000 7f0fff04f640 1 -- 172.21.15.64:0/2887060434 >> v1:172.21.15.64:6790/0 conn(0x7f0ff8135f70 legacy=0x7f0ff81415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.442+0000 7f0fff04f640 1 -- 172.21.15.64:0/2887060434 shutdown_connections 2024-09-17T15:54:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.442+0000 7f0fff04f640 1 -- 172.21.15.64:0/2887060434 >> 172.21.15.64:0/2887060434 conn(0x7f0ff806db30 msgr2=0x7f0ff80ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.442+0000 7f0fff04f640 1 -- 172.21.15.64:0/2887060434 shutdown_connections 2024-09-17T15:54:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.442+0000 7f0fff04f640 1 -- 172.21.15.64:0/2887060434 wait complete. 2024-09-17T15:54:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.442+0000 7f0fff04f640 10 monclient: build_initial_monmap 2024-09-17T15:54:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.442+0000 7f0fff04f640 10 monclient: monmap: 2024-09-17T15:54:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:13.458 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0fff04f640 1 Processor -- start 2024-09-17T15:54:13.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0fff04f640 1 -- start start 2024-09-17T15:54:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0fff04f640 10 monclient: init 2024-09-17T15:54:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0fff04f640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0fff04f640 10 monclient: _add_conns ranks=[4,1,8,2,3,0,6,5,7] 2024-09-17T15:54:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0fff04f640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff80a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0fff04f640 10 monclient(hunting): picked mon.noname-f con 0x7f0ff8135b50 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0fff04f640 10 monclient(hunting): picked mon.noname-e con 0x7f0ff8135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0fff04f640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ff8134920 con 0x7f0ff8135b50 2024-09-17T15:54:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0fff04f640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ff81321e0 con 0x7f0ff80a5520 2024-09-17T15:54:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0fff04f640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ff80a28d0 con 0x7f0ff8135f70 2024-09-17T15:54:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0fff04f640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0fff04f640 10 monclient(hunting): authenticate will time out at 1251.118877s 2024-09-17T15:54:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0ffcdeb640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f0ff80a5520 0x7f0ff809f790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:48238/0 (socket says 172.21.15.64:48238) 2024-09-17T15:54:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.443+0000 7f0ffcdeb640 1 -- 172.21.15.64:0/441129617 learned_addr learned my addr 172.21.15.64:0/441129617 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2926606168 0 0) 0x7f0ff81321e0 con 0x7f0ff80a5520 2024-09-17T15:54:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): my global_id is 154669 2024-09-17T15:54:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fdc003520 con 0x7f0ff80a5520 2024-09-17T15:54:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4007366477 0 0) 0x7f0ff80a28d0 con 0x7f0ff8135f70 2024-09-17T15:54:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): my global_id is 174149 2024-09-17T15:54:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ff81321e0 con 0x7f0ff8135f70 2024-09-17T15:54:13.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4138158191 0 0) 0x7f0ff8134920 con 0x7f0ff8135b50 2024-09-17T15:54:13.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): my global_id is 6940 2024-09-17T15:54:13.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ff80a28d0 con 0x7f0ff8135b50 2024-09-17T15:54:13.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 219182680 0 0) 0x7f0fdc003520 con 0x7f0ff80a5520 2024-09-17T15:54:13.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ff8134920 con 0x7f0ff80a5520 2024-09-17T15:54:13.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0fec001bb0 con 0x7f0ff80a5520 2024-09-17T15:54:13.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:13.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): dump: 2024-09-17T15:54:13.462 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1207562881 0 0) 0x7f0ff81321e0 con 0x7f0ff8135f70 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fdc003520 con 0x7f0ff8135f70 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0ff0001f80 con 0x7f0ff8135f70 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): dump: 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.464 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.465 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.465 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.465 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.465 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.465 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.465 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.465 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.465 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.465 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.465 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.465 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.465 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3921344910 0 0) 0x7f0ff80a28d0 con 0x7f0ff8135b50 2024-09-17T15:54:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ff81321e0 con 0x7f0ff8135b50 2024-09-17T15:54:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0fe4001a10 con 0x7f0ff8135b50 2024-09-17T15:54:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:54:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient(hunting): dump: 2024-09-17T15:54:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.467 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.467 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.467 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.467 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.467 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.467 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.467 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.467 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 941760002 0 0) 0x7f0ff8134920 con 0x7f0ff80a5520 2024-09-17T15:54:13.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 >> v1:172.21.15.64:6793/0 conn(0x7f0ff8135f70 legacy=0x7f0ff81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 >> v1:172.21.15.100:6789/0 conn(0x7f0ff8135b50 legacy=0x7f0ff809fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient: _finish_hunting 0 2024-09-17T15:54:13.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 monclient: found mon.c 2024-09-17T15:54:13.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:13.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:13.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff80a17f0 con 0x7f0ff80a5520 2024-09-17T15:54:13.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient: _finish_auth 0 2024-09-17T15:54:13.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:54:13.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:13.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0fff04f640 5 monclient: authenticate success, global_id 154669 2024-09-17T15:54:13.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0fff04f640 10 monclient: _renew_subs 2024-09-17T15:54:13.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0fff04f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:13.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0fff04f640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff80a13b0 con 0x7f0ff80a5520 2024-09-17T15:54:13.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0fff04f640 10 monclient: _renew_subs 2024-09-17T15:54:13.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0fff04f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:13.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0fff04f640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f0ff80a05b0 con 0x7f0ff80a5520 2024-09-17T15:54:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0fec0038c0 con 0x7f0ff80a5520 2024-09-17T15:54:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0fec003ef0 con 0x7f0ff80a5520 2024-09-17T15:54:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff6ffd640 4 set_mon_vals no callback set 2024-09-17T15:54:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.444+0000 7f0ff57fa640 10 monclient: dump: 2024-09-17T15:54:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.445+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f0fec004110 con 0x7f0ff80a5520 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.445+0000 7f0fff04f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.445+0000 7f0fff04f640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0fd80065c0 con 0x7f0fd8002860 2024-09-17T15:54:13.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.445+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f0fec04eab0 con 0x7f0ff80a5520 2024-09-17T15:54:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ffd5ec640 10 monclient: get_auth_request con 0x7f0fdc040d30 auth_method 0 2024-09-17T15:54:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1454909464 0 0) 0x7f0fd80065c0 con 0x7f0fd8002860 2024-09-17T15:54:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 10 monclient(hunting): my global_id is 115923 2024-09-17T15:54:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ff8134920 con 0x7f0fd8002860 2024-09-17T15:54:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1967078196 0 0) 0x7f0ff8134920 con 0x7f0fd8002860 2024-09-17T15:54:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fd80065c0 con 0x7f0fd8002860 2024-09-17T15:54:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0fe4003930 con 0x7f0fd8002860 2024-09-17T15:54:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:54:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 10 monclient: dump: 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.473 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.473 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2155424601 0 0) 0x7f0fd80065c0 con 0x7f0fd8002860 2024-09-17T15:54:13.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.446+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.100:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0fd800a160 con 0x7f0fd8002860 2024-09-17T15:54:13.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.447+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0fe4005b50 con 0x7f0fd8002860 2024-09-17T15:54:13.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.447+0000 7f0ff57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:13.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.447+0000 7f0ff57fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:13.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.447+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 >> v1:172.21.15.100:6790/0 conn(0x7f0fd8002860 legacy=0x7f0fd8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.447+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 >> v1:172.21.15.100:6790/0 conn(0x7f0fd8002860 legacy=0x7f0fd8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:13.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0fff04f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:54:13.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0fff04f640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0fd8009e20 con 0x7f0fd80090e0 2024-09-17T15:54:13.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1989623392 0 0) 0x7f0fd8009e20 con 0x7f0fd80090e0 2024-09-17T15:54:13.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 10 monclient(hunting): my global_id is 115932 2024-09-17T15:54:13.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fd80065c0 con 0x7f0fd80090e0 2024-09-17T15:54:13.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 769710398 0 0) 0x7f0fd80065c0 con 0x7f0fd80090e0 2024-09-17T15:54:13.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fd8009e20 con 0x7f0fd80090e0 2024-09-17T15:54:13.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0fe4005480 con 0x7f0fd80090e0 2024-09-17T15:54:13.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:54:13.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 10 monclient: dump: 2024-09-17T15:54:13.476 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.476 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.476 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.476 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.476 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.476 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.476 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.476 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.476 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.477 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.477 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.477 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.477 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.477 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.477 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.477 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2012545189 0 0) 0x7f0fd8009e20 con 0x7f0fd80090e0 2024-09-17T15:54:13.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.453+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 --> v1:172.21.15.100:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0fd800c3f0 con 0x7f0fd80090e0 2024-09-17T15:54:13.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 <== mon.3 v1:172.21.15.100:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 772132015) 0x7f0fe4005570 con 0x7f0fd80090e0 2024-09-17T15:54:13.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0ff57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:54:13.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0ff57fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:13.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 >> v1:172.21.15.100:6790/0 conn(0x7f0fd80090e0 legacy=0x7f0fd8009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0ff57fa640 1 -- 172.21.15.64:0/441129617 >> v1:172.21.15.100:6790/0 conn(0x7f0fd80090e0 legacy=0x7f0fd8009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:13.478 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:54:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:54:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:54:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 46, 2024-09-17T15:54:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 376677, 2024-09-17T15:54:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:54:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:54:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:54:13.478 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:54:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:54:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:54:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:54:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:54:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:54:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:54:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:54:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:54:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:54:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:54:13.480 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.481 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:54:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:54:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:54:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:54:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:54:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:54:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:54:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:54:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:54:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:54:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:54:13.482 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.483 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:54:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:54:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:54:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:54:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:54:13.484 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.485 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:54:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:54:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:54:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:54:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:54:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:54:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.487 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:54:13.488 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:54:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:54:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.489 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:54:13.492 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:54:13.492 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.492 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.492 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.492 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:54:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:54:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:54:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:54:13.494 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:54:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:54:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.495 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:54:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:54:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:54:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:54:13.496 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.498 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": [ 2024-09-17T15:54:13.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 2 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:54:13.500 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:13.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0fff04f640 1 -- 172.21.15.64:0/441129617 >> v1:172.21.15.100:6812/4210963484 conn(0x7f0fdc040d30 legacy=0x7f0fdc0431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0fff04f640 10 monclient: shutdown 2024-09-17T15:54:13.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0fff04f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:13.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0fff04f640 1 -- 172.21.15.64:0/441129617 >> v1:172.21.15.64:6791/0 conn(0x7f0ff80a5520 legacy=0x7f0ff809f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0ffd5ec640 1 -- 172.21.15.64:0/441129617 reap_dead start 2024-09-17T15:54:13.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0ffd5ec640 1 -- 172.21.15.64:0/441129617 reap_dead start 2024-09-17T15:54:13.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0fff04f640 1 -- 172.21.15.64:0/441129617 shutdown_connections 2024-09-17T15:54:13.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0fff04f640 1 -- 172.21.15.64:0/441129617 >> 172.21.15.64:0/441129617 conn(0x7f0ff806db30 msgr2=0x7f0ff81334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:13.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0fff04f640 1 -- 172.21.15.64:0/441129617 shutdown_connections 2024-09-17T15:54:13.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.454+0000 7f0fff04f640 1 -- 172.21.15.64:0/441129617 wait complete. 2024-09-17T15:54:13.503 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:54:13.503 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-17T15:54:13.503 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-17T15:54:13.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.603+0000 7f0b4a1bb640 10 monclient: get_monmap_and_config 2024-09-17T15:54:13.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.603+0000 7f0b4a1bb640 10 monclient: build_initial_monmap 2024-09-17T15:54:13.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.603+0000 7f0b4a1bb640 10 monclient: monmap: 2024-09-17T15:54:13.605 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:13.605 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.605 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:13.604486+0000 2024-09-17T15:54:13.605 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:13.604486+0000 2024-09-17T15:54:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:13.606 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.604+0000 7f0b4a1bb640 1 Processor -- start 2024-09-17T15:54:13.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.604+0000 7f0b4a1bb640 1 -- start start 2024-09-17T15:54:13.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.604+0000 7f0b4a1bb640 10 monclient: init 2024-09-17T15:54:13.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.604+0000 7f0b4a1bb640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:13.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.604+0000 7f0b4a1bb640 10 monclient: _add_conns ranks=[8,3,6,1,2,7,0,4,5] 2024-09-17T15:54:13.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.604+0000 7f0b4a1bb640 10 monclient(hunting): picked mon.noname-e con 0x7f0b440a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:13.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b4a1bb640 10 monclient(hunting): picked mon.noname-g con 0x7f0b44135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:13.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b4a1bb640 10 monclient(hunting): picked mon.noname-d con 0x7f0b44135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:13.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b4a1bb640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b4409d6f0 con 0x7f0b44135b50 2024-09-17T15:54:13.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b4a1bb640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b4409d9d0 con 0x7f0b44135f70 2024-09-17T15:54:13.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b4a1bb640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b4409dcb0 con 0x7f0b440a5520 2024-09-17T15:54:13.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b4a1bb640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:13.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b4a1bb640 10 monclient(hunting): authenticate will time out at 1251.280129s 2024-09-17T15:54:13.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b437fe640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f0b440a5520 0x7f0b440a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51308/0 (socket says 172.21.15.64:51308) 2024-09-17T15:54:13.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b437fe640 1 -- 172.21.15.64:0/896618509 learned_addr learned my addr 172.21.15.64:0/896618509 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:13.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 1 -- 172.21.15.64:0/896618509 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 202025883 0 0) 0x7f0b4409d9d0 con 0x7f0b44135f70 2024-09-17T15:54:13.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient(hunting): my global_id is 174183 2024-09-17T15:54:13.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 1 -- 172.21.15.64:0/896618509 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b2c003520 con 0x7f0b44135f70 2024-09-17T15:54:13.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 1 -- 172.21.15.64:0/896618509 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1743031206 0 0) 0x7f0b4409dcb0 con 0x7f0b440a5520 2024-09-17T15:54:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient(hunting): my global_id is 174158 2024-09-17T15:54:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 1 -- 172.21.15.64:0/896618509 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b4409d9d0 con 0x7f0b440a5520 2024-09-17T15:54:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 1 -- 172.21.15.64:0/896618509 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1146725336 0 0) 0x7f0b2c003520 con 0x7f0b44135f70 2024-09-17T15:54:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 1 -- 172.21.15.64:0/896618509 >> v1:172.21.15.64:6793/0 conn(0x7f0b440a5520 legacy=0x7f0b440a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 1 -- 172.21.15.64:0/896618509 >> v1:172.21.15.100:6790/0 conn(0x7f0b44135b50 legacy=0x7f0b4413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient: _finish_hunting 0 2024-09-17T15:54:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 1 monclient: found mon.noname-d 2024-09-17T15:54:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:54:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 1 -- 172.21.15.64:0/896618509 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4409ccd0 con 0x7f0b44135f70 2024-09-17T15:54:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient: _finish_auth 0 2024-09-17T15:54:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient: _check_auth_tickets 2024-09-17T15:54:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 1 -- 172.21.15.64:0/896618509 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0b3c001bb0 con 0x7f0b44135f70 2024-09-17T15:54:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:54:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient: dump: 2024-09-17T15:54:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.619 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.619 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.619 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.619 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.619 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.619 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.619 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.619 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.619 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.619 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.620 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b4a1bb640 5 monclient: authenticate success, global_id 174183 2024-09-17T15:54:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b4a1bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 1 -- 172.21.15.64:0/896618509 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0b3c001d50 con 0x7f0b44135f70 2024-09-17T15:54:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 1 -- 172.21.15.64:0/896618509 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0b3c003cd0 con 0x7f0b44135f70 2024-09-17T15:54:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b4a1bb640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b4a1bb640 4 set_mon_vals no callback set 2024-09-17T15:54:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.605+0000 7f0b427fc640 10 monclient: dump: 2024-09-17T15:54:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.621 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.606+0000 7f0b4a1bb640 10 monclient: shutdown 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.606+0000 7f0b4a1bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.606+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/896618509 >> v1:172.21.15.64:6792/0 conn(0x7f0b44135f70 legacy=0x7f0b441415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.606+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/896618509 shutdown_connections 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.606+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/896618509 >> 172.21.15.64:0/896618509 conn(0x7f0b4406db30 msgr2=0x7f0b440ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.606+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/896618509 shutdown_connections 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.606+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/896618509 wait complete. 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.606+0000 7f0b4a1bb640 10 monclient: build_initial_monmap 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.607+0000 7f0b4a1bb640 10 monclient: monmap: 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:13.623 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.607+0000 7f0b4a1bb640 1 Processor -- start 2024-09-17T15:54:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.607+0000 7f0b4a1bb640 1 -- start start 2024-09-17T15:54:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.607+0000 7f0b4a1bb640 10 monclient: init 2024-09-17T15:54:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.607+0000 7f0b4a1bb640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.607+0000 7f0b4a1bb640 10 monclient: _add_conns ranks=[6,8,5,3,0,7,2,4,1] 2024-09-17T15:54:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b4a1bb640 10 monclient(hunting): picked mon.noname-d con 0x7f0b440a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b4a1bb640 10 monclient(hunting): picked mon.noname-e con 0x7f0b44135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b4a1bb640 10 monclient(hunting): picked mon.noname-h con 0x7f0b44135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b4a1bb640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b440a8f70 con 0x7f0b44135f70 2024-09-17T15:54:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b4a1bb640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b44135310 con 0x7f0b440a5520 2024-09-17T15:54:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b4a1bb640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b44134e00 con 0x7f0b44135b50 2024-09-17T15:54:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b4a1bb640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b4a1bb640 10 monclient(hunting): authenticate will time out at 1251.283141s 2024-09-17T15:54:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b42ffd640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f0b44135b50 0x7f0b440a4940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51318/0 (socket says 172.21.15.64:51318) 2024-09-17T15:54:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b437fe640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f0b440a5520 0x7f0b441cd4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:37252/0 (socket says 172.21.15.64:37252) 2024-09-17T15:54:13.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b42ffd640 1 -- 172.21.15.64:0/1719062364 learned_addr learned my addr 172.21.15.64:0/1719062364 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:13.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4198559059 0 0) 0x7f0b44134e00 con 0x7f0b44135b50 2024-09-17T15:54:13.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): my global_id is 174167 2024-09-17T15:54:13.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b30003550 con 0x7f0b44135b50 2024-09-17T15:54:13.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2085859629 0 0) 0x7f0b44135310 con 0x7f0b440a5520 2024-09-17T15:54:13.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): my global_id is 174192 2024-09-17T15:54:13.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b44134e00 con 0x7f0b440a5520 2024-09-17T15:54:13.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1846398013 0 0) 0x7f0b4409ccd0 con 0x7f0b44135f70 2024-09-17T15:54:13.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): my global_id is 174209 2024-09-17T15:54:13.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b44135310 con 0x7f0b44135f70 2024-09-17T15:54:13.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 917442464 0 0) 0x7f0b44134e00 con 0x7f0b440a5520 2024-09-17T15:54:13.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b4409ccd0 con 0x7f0b440a5520 2024-09-17T15:54:13.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0b34001550 con 0x7f0b440a5520 2024-09-17T15:54:13.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:54:13.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): dump: 2024-09-17T15:54:13.626 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.627 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4125140537 0 0) 0x7f0b30003550 con 0x7f0b44135b50 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b44134e00 con 0x7f0b44135b50 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0b38001b60 con 0x7f0b44135b50 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): dump: 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.628 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.629 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3645618469 0 0) 0x7f0b44135310 con 0x7f0b44135f70 2024-09-17T15:54:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b30003550 con 0x7f0b44135f70 2024-09-17T15:54:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0b3c001af0 con 0x7f0b44135f70 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient(hunting): dump: 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.6 v1:172.21.15.64:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3142257919 0 0) 0x7f0b4409ccd0 con 0x7f0b440a5520 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 >> v1:172.21.15.64:6793/0 conn(0x7f0b44135b50 legacy=0x7f0b440a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 >> v1:172.21.15.100:6791/0 conn(0x7f0b44135f70 legacy=0x7f0b441d5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 monclient: found mon.d 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b44132fa0 con 0x7f0b440a5520 2024-09-17T15:54:13.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient: _finish_auth 0 2024-09-17T15:54:13.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:54:13.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:13.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b4a1bb640 5 monclient: authenticate success, global_id 174192 2024-09-17T15:54:13.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b4a1bb640 10 monclient: _renew_subs 2024-09-17T15:54:13.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b4a1bb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:54:13.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.64:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0b440a0100 con 0x7f0b440a5520 2024-09-17T15:54:13.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.608+0000 7f0b4a1bb640 10 monclient: _renew_subs 2024-09-17T15:54:13.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.609+0000 7f0b4a1bb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.64:6792/0 2024-09-17T15:54:13.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.609+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.64:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f0b440a5050 con 0x7f0b440a5520 2024-09-17T15:54:13.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.609+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.6 v1:172.21.15.64:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0b34001770 con 0x7f0b440a5520 2024-09-17T15:54:13.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.609+0000 7f0b2bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.609+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.6 v1:172.21.15.64:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0b34004dd0 con 0x7f0b440a5520 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.609+0000 7f0b2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.609+0000 7f0b2bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.609+0000 7f0b41ffb640 4 set_mon_vals no callback set 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.609+0000 7f0b2bfff640 10 monclient: dump: 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.633 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.609+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.6 v1:172.21.15.64:6792/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f0b340116b0 con 0x7f0b440a5520 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.609+0000 7f0b4a1bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.609+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0b240065c0 con 0x7f0b24002860 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.6 v1:172.21.15.64:6792/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f0b3404e980 con 0x7f0b440a5520 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b43fff640 10 monclient: get_auth_request con 0x7f0b30040d50 auth_method 0 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2907185080 0 0) 0x7f0b240065c0 con 0x7f0b24002860 2024-09-17T15:54:13.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 10 monclient(hunting): my global_id is 174218 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b4409ccd0 con 0x7f0b24002860 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4005877373 0 0) 0x7f0b4409ccd0 con 0x7f0b24002860 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b240065c0 con 0x7f0b24002860 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0b38003c80 con 0x7f0b24002860 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 10 monclient: dump: 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.635 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2555276638 0 0) 0x7f0b240065c0 con 0x7f0b24002860 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.610+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.100:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0b2400a160 con 0x7f0b24002860 2024-09-17T15:54:13.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.611+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0b38003f30 con 0x7f0b24002860 2024-09-17T15:54:13.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.611+0000 7f0b2bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:13.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.611+0000 7f0b2bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:13.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.611+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 >> v1:172.21.15.100:6791/0 conn(0x7f0b24002860 legacy=0x7f0b24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.611+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 >> v1:172.21.15.100:6791/0 conn(0x7f0b24002860 legacy=0x7f0b24004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:13.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.617+0000 7f0b4a1bb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:54:13.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.617+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0b24009e30 con 0x7f0b24009110 2024-09-17T15:54:13.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.617+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 646999884 0 0) 0x7f0b24009e30 con 0x7f0b24009110 2024-09-17T15:54:13.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.617+0000 7f0b2bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.617+0000 7f0b2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.617+0000 7f0b2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.617+0000 7f0b2bfff640 10 monclient(hunting): my global_id is 174227 2024-09-17T15:54:13.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.617+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b240065c0 con 0x7f0b24009110 2024-09-17T15:54:13.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1954944827 0 0) 0x7f0b240065c0 con 0x7f0b24009110 2024-09-17T15:54:13.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b24009e30 con 0x7f0b24009110 2024-09-17T15:54:13.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0b380043d0 con 0x7f0b24009110 2024-09-17T15:54:13.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b2bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-17T15:54:13.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b2bfff640 10 monclient: dump: 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.640 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.641 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.641 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.5 v1:172.21.15.100:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2226557223 0 0) 0x7f0b24009e30 con 0x7f0b24009110 2024-09-17T15:54:13.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 --> v1:172.21.15.100:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0b2400c3f0 con 0x7f0b24009110 2024-09-17T15:54:13.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 <== mon.5 v1:172.21.15.100:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 3018373314) 0x7f0b38004400 con 0x7f0b24009110 2024-09-17T15:54:13.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b2bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:54:13.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b2bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:13.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 >> v1:172.21.15.100:6791/0 conn(0x7f0b24009110 legacy=0x7f0b24009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b2bfff640 1 -- 172.21.15.64:0/1719062364 >> v1:172.21.15.100:6791/0 conn(0x7f0b24009110 legacy=0x7f0b24009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:13.642 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:54:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:54:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:54:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 46, 2024-09-17T15:54:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 16575, 2024-09-17T15:54:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:54:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:54:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:54:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:54:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:54:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.643 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:54:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:54:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.644 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:54:13.645 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.646 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:54:13.647 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:54:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:54:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:54:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:54:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.648 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:54:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:54:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:54:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:54:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.650 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:54:13.651 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:54:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:54:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:54:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:54:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.652 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:54:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:54:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:54:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.653 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:54:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:54:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:54:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.655 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.656 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:54:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:54:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:54:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:54:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:54:13.659 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.661 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:54:13.661 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:13.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/1719062364 >> v1:172.21.15.100:6812/4210963484 conn(0x7f0b30040d50 legacy=0x7f0b30043210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b4a1bb640 10 monclient: shutdown 2024-09-17T15:54:13.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b4a1bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:13.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/1719062364 >> v1:172.21.15.64:6792/0 conn(0x7f0b440a5520 legacy=0x7f0b441cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.618+0000 7f0b43fff640 1 -- 172.21.15.64:0/1719062364 reap_dead start 2024-09-17T15:54:13.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.619+0000 7f0b43fff640 1 -- 172.21.15.64:0/1719062364 reap_dead start 2024-09-17T15:54:13.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.619+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/1719062364 shutdown_connections 2024-09-17T15:54:13.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.619+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/1719062364 >> 172.21.15.64:0/1719062364 conn(0x7f0b4406db30 msgr2=0x7f0b441334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:13.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.619+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/1719062364 shutdown_connections 2024-09-17T15:54:13.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.619+0000 7f0b4a1bb640 1 -- 172.21.15.64:0/1719062364 wait complete. 2024-09-17T15:54:13.663 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:54:13.663 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-17T15:54:13.663 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-17T15:54:13.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.763+0000 7f6a2fe2a640 10 monclient: get_monmap_and_config 2024-09-17T15:54:13.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.763+0000 7f6a2fe2a640 10 monclient: build_initial_monmap 2024-09-17T15:54:13.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.763+0000 7f6a2fe2a640 10 monclient: monmap: 2024-09-17T15:54:13.765 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:13.765 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.765 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:13.764278+0000 2024-09-17T15:54:13.765 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:13.764278+0000 2024-09-17T15:54:13.765 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:13.765 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.766 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:13.766 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:13.766 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:13.766 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:13.766 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:13.766 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:13.766 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:13.766 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:13.766 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:13.766 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2fe2a640 1 Processor -- start 2024-09-17T15:54:13.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2fe2a640 1 -- start start 2024-09-17T15:54:13.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2fe2a640 10 monclient: init 2024-09-17T15:54:13.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2fe2a640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:13.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2fe2a640 10 monclient: _add_conns ranks=[2,5,4,0,8,7,1,3,6] 2024-09-17T15:54:13.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2fe2a640 10 monclient(hunting): picked mon.noname-b con 0x7f6a280a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:13.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2fe2a640 10 monclient(hunting): picked mon.noname-h con 0x7f6a28135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:13.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2fe2a640 10 monclient(hunting): picked mon.noname-c con 0x7f6a28135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:13.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2fe2a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a2809d6f0 con 0x7f6a280a5520 2024-09-17T15:54:13.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2fe2a640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a2809d9d0 con 0x7f6a28135f70 2024-09-17T15:54:13.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2fe2a640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a2809dcb0 con 0x7f6a28135b50 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2fe2a640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2fe2a640 10 monclient(hunting): authenticate will time out at 1251.439770s 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2e3c7640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f6a28135f70 0x7f6a281415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:48242/0 (socket says 172.21.15.64:48242) 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.764+0000 7f6a2e3c7640 1 -- 172.21.15.64:0/3734652791 learned_addr learned my addr 172.21.15.64:0/3734652791 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 1 -- 172.21.15.64:0/3734652791 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1963675333 0 0) 0x7f6a2809d9d0 con 0x7f6a28135f70 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient(hunting): my global_id is 154678 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 1 -- 172.21.15.64:0/3734652791 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a14003520 con 0x7f6a28135f70 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 1 -- 172.21.15.64:0/3734652791 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3994371718 0 0) 0x7f6a2809d6f0 con 0x7f6a280a5520 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient(hunting): my global_id is 125255 2024-09-17T15:54:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 1 -- 172.21.15.64:0/3734652791 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a2809d9d0 con 0x7f6a280a5520 2024-09-17T15:54:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 1 -- 172.21.15.64:0/3734652791 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3761984680 0 0) 0x7f6a2809d9d0 con 0x7f6a280a5520 2024-09-17T15:54:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 1 -- 172.21.15.64:0/3734652791 >> v1:172.21.15.100:6791/0 conn(0x7f6a28135b50 legacy=0x7f6a2813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 1 -- 172.21.15.64:0/3734652791 >> v1:172.21.15.64:6791/0 conn(0x7f6a28135f70 legacy=0x7f6a281415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient: _finish_hunting 0 2024-09-17T15:54:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 1 monclient: found mon.noname-b 2024-09-17T15:54:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:54:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 1 -- 172.21.15.64:0/3734652791 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2809ccd0 con 0x7f6a280a5520 2024-09-17T15:54:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient: _finish_auth 0 2024-09-17T15:54:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient: _check_auth_tickets 2024-09-17T15:54:13.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2fe2a640 5 monclient: authenticate success, global_id 125255 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 1 -- 172.21.15.64:0/3734652791 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6a24001810 con 0x7f6a280a5520 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2fe2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient: dump: 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.777 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2fe2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 1 -- 172.21.15.64:0/3734652791 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a240019b0 con 0x7f6a280a5520 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 1 -- 172.21.15.64:0/3734652791 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6a24003950 con 0x7f6a280a5520 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2fe2a640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:13.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2fe2a640 4 set_mon_vals no callback set 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2cbc4640 10 monclient: dump: 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.779 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.780 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.780 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.780 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.780 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2fe2a640 10 monclient: shutdown 2024-09-17T15:54:13.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2fe2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:13.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.765+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/3734652791 >> v1:172.21.15.64:6790/0 conn(0x7f6a280a5520 legacy=0x7f6a280a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.766+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/3734652791 shutdown_connections 2024-09-17T15:54:13.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.766+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/3734652791 >> 172.21.15.64:0/3734652791 conn(0x7f6a2806db30 msgr2=0x7f6a280ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:13.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.766+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/3734652791 shutdown_connections 2024-09-17T15:54:13.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.766+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/3734652791 wait complete. 2024-09-17T15:54:13.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.766+0000 7f6a2fe2a640 10 monclient: build_initial_monmap 2024-09-17T15:54:13.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.766+0000 7f6a2fe2a640 10 monclient: monmap: 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:13.781 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.766+0000 7f6a2fe2a640 1 Processor -- start 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2fe2a640 1 -- start start 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2fe2a640 10 monclient: init 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2fe2a640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2fe2a640 10 monclient: _add_conns ranks=[0,5,8,4,3,2,6,7,1] 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2fe2a640 10 monclient(hunting): picked mon.noname-a con 0x7f6a280a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2fe2a640 10 monclient(hunting): picked mon.noname-h con 0x7f6a28135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2fe2a640 10 monclient(hunting): picked mon.noname-e con 0x7f6a28135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2fe2a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a28135310 con 0x7f6a280a5520 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2fe2a640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a28134f50 con 0x7f6a28135b50 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2fe2a640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a281389e0 con 0x7f6a28135f70 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2fe2a640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:13.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2fe2a640 10 monclient(hunting): authenticate will time out at 1251.442501s 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2e3c7640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f6a28135f70 0x7f6a281d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51322/0 (socket says 172.21.15.64:51322) 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a2e3c7640 1 -- 172.21.15.64:0/2497971968 learned_addr learned my addr 172.21.15.64:0/2497971968 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3681254903 0 0) 0x7f6a281389e0 con 0x7f6a28135f70 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 10 monclient(hunting): my global_id is 174176 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a0c003550 con 0x7f6a28135f70 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3231628354 0 0) 0x7f6a28135310 con 0x7f6a280a5520 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 10 monclient(hunting): my global_id is 125136 2024-09-17T15:54:13.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a281389e0 con 0x7f6a280a5520 2024-09-17T15:54:13.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4110376137 0 0) 0x7f6a28134f50 con 0x7f6a28135b50 2024-09-17T15:54:13.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 10 monclient(hunting): my global_id is 174245 2024-09-17T15:54:13.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.767+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a28135310 con 0x7f6a28135b50 2024-09-17T15:54:13.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1347280579 0 0) 0x7f6a0c003550 con 0x7f6a28135f70 2024-09-17T15:54:13.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a28134f50 con 0x7f6a28135f70 2024-09-17T15:54:13.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6a20001ce0 con 0x7f6a28135f70 2024-09-17T15:54:13.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient(hunting): dump: 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.785 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2198708333 0 0) 0x7f6a281389e0 con 0x7f6a280a5520 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a0c003550 con 0x7f6a280a5520 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6a24003930 con 0x7f6a280a5520 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient(hunting): dump: 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.786 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3634434790 0 0) 0x7f6a28135310 con 0x7f6a28135b50 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a281389e0 con 0x7f6a28135b50 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6a18001990 con 0x7f6a28135b50 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient(hunting): dump: 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1300060351 0 0) 0x7f6a28134f50 con 0x7f6a28135f70 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 >> v1:172.21.15.100:6791/0 conn(0x7f6a28135b50 legacy=0x7f6a28137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 >> v1:172.21.15.64:6789/0 conn(0x7f6a280a5520 legacy=0x7f6a28136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 monclient: found mon.e 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a28137d60 con 0x7f6a28135f70 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a2fe2a640 5 monclient: authenticate success, global_id 174176 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a2fe2a640 10 monclient: _renew_subs 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a2fe2a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a2813ad40 con 0x7f6a28135f70 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a2fe2a640 10 monclient: _renew_subs 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a2fe2a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a2813b160 con 0x7f6a28135f70 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a20002000 con 0x7f6a28135f70 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:13.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6a20005150 con 0x7f6a28135f70 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1e7fc640 10 monclient: dump: 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.791 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.768+0000 7f6a1ffff640 4 set_mon_vals no callback set 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.769+0000 7f6a2fe2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.769+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6a100065c0 con 0x7f6a10002860 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.769+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f6a20011ab0 con 0x7f6a28135f70 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.769+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f6a2004ed20 con 0x7f6a28135f70 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.769+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1992488796 0 0) 0x7f6a100065c0 con 0x7f6a10002860 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.769+0000 7f6a1e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.769+0000 7f6a1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.769+0000 7f6a1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.769+0000 7f6a1e7fc640 10 monclient(hunting): my global_id is 174148 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a28134f50 con 0x7f6a10002860 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a2d3c5640 10 monclient: get_auth_request con 0x7f6a0c040d70 auth_method 0 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2766501236 0 0) 0x7f6a28134f50 con 0x7f6a10002860 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a100065c0 con 0x7f6a10002860 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6a24003c10 con 0x7f6a10002860 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 10 monclient: dump: 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.793 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2332779379 0 0) 0x7f6a100065c0 con 0x7f6a10002860 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.100:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a1000a160 con 0x7f6a10002860 2024-09-17T15:54:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6a24006480 con 0x7f6a10002860 2024-09-17T15:54:13.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:13.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:13.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 >> v1:172.21.15.100:6792/0 conn(0x7f6a10002860 legacy=0x7f6a10004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.770+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 >> v1:172.21.15.100:6792/0 conn(0x7f6a10002860 legacy=0x7f6a10004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:13.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.776+0000 7f6a2fe2a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:54:13.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.776+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6a10009e00 con 0x7f6a100090e0 2024-09-17T15:54:13.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.777+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 190002219 0 0) 0x7f6a10009e00 con 0x7f6a100090e0 2024-09-17T15:54:13.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.777+0000 7f6a1e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.777+0000 7f6a1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.777+0000 7f6a1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.777+0000 7f6a1e7fc640 10 monclient(hunting): my global_id is 174157 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.777+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a100065c0 con 0x7f6a100090e0 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.777+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 64364641 0 0) 0x7f6a100065c0 con 0x7f6a100090e0 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.777+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a10009e00 con 0x7f6a100090e0 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.777+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6a24004630 con 0x7f6a100090e0 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.777+0000 7f6a1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.777+0000 7f6a1e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.777+0000 7f6a1e7fc640 10 monclient: dump: 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.7 v1:172.21.15.100:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3719157099 0 0) 0x7f6a10009e00 con 0x7f6a100090e0 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 --> v1:172.21.15.100:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6a1000c3a0 con 0x7f6a100090e0 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 <== mon.7 v1:172.21.15.100:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 127352942) 0x7f6a24005b40 con 0x7f6a100090e0 2024-09-17T15:54:13.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a1e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:54:13.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a1e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:13.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 >> v1:172.21.15.100:6792/0 conn(0x7f6a100090e0 legacy=0x7f6a10009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a1e7fc640 1 -- 172.21.15.64:0/2497971968 >> v1:172.21.15.100:6792/0 conn(0x7f6a100090e0 legacy=0x7f6a10009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:13.799 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:54:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:54:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "peon", 2024-09-17T15:54:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "election_epoch": 46, 2024-09-17T15:54:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 16735, 2024-09-17T15:54:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum": [ 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: 0, 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: 5, 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_age": 1, 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "required_con": "2450521708007866372", 2024-09-17T15:54:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "required_mon": [ 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:54:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "quorum_mon": [ 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "outside_quorum": [], 2024-09-17T15:54:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "extra_probe_peers": [], 2024-09-17T15:54:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "sync_provider": [], 2024-09-17T15:54:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": { 2024-09-17T15:54:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 1, 2024-09-17T15:54:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "3de99658-3189-432a-9efe-6c9d4c0a2934", 2024-09-17T15:54:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-17T15:45:15.994818Z", 2024-09-17T15:54:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release": 19, 2024-09-17T15:54:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "min_mon_release_name": "squid", 2024-09-17T15:54:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "election_strategy": 3, 2024-09-17T15:54:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "disallowed_leaders: ": "", 2024-09-17T15:54:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false, 2024-09-17T15:54:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "tiebreaker_mon": "", 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_ranks: ": "", 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "features": { 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "persistent": [ 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "kraken", 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "luminous", 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "mimic", 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap-prune", 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "nautilus", 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus", 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific", 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "elector-pinging", 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy", 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "reef", 2024-09-17T15:54:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "squid" 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "optional": [] 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "mons": [ 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 0, 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "a", 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789", 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6789/0", 2024-09-17T15:54:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6789/0", 2024-09-17T15:54:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 1, 2024-09-17T15:54:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "f", 2024-09-17T15:54:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789", 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6789/0", 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6789/0", 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 2, 2024-09-17T15:54:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "b", 2024-09-17T15:54:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790", 2024-09-17T15:54:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6790/0", 2024-09-17T15:54:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6790/0", 2024-09-17T15:54:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 3, 2024-09-17T15:54:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "g", 2024-09-17T15:54:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790", 2024-09-17T15:54:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6790/0", 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6790/0", 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 4, 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "c", 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791", 2024-09-17T15:54:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6791/0", 2024-09-17T15:54:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6791/0", 2024-09-17T15:54:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 5, 2024-09-17T15:54:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "h", 2024-09-17T15:54:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791", 2024-09-17T15:54:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6791/0", 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6791/0", 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 6, 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "d", 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792", 2024-09-17T15:54:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6792/0", 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6792/0", 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 7, 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "i", 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792", 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.100:6792/0", 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.100:6792/0", 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "rank": 8, 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "e", 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793", 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 0 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6793/0", 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6793/0", 2024-09-17T15:54:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "priority": 0, 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "{}" 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "feature_map": { 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "mon": [ 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-17T15:54:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client": [ 2024-09-17T15:54:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-17T15:54:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:54:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "release": "squid", 2024-09-17T15:54:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 1 2024-09-17T15:54:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-17T15:54:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-17T15:54:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-17T15:54:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": false 2024-09-17T15:54:13.818 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:13.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/2497971968 >> v1:172.21.15.100:6812/4210963484 conn(0x7f6a0c040d70 legacy=0x7f6a0c043230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a2fe2a640 10 monclient: shutdown 2024-09-17T15:54:13.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a2fe2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:13.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/2497971968 >> v1:172.21.15.64:6793/0 conn(0x7f6a28135f70 legacy=0x7f6a281d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a2e3c7640 1 -- 172.21.15.64:0/2497971968 reap_dead start 2024-09-17T15:54:13.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a2e3c7640 1 -- 172.21.15.64:0/2497971968 reap_dead start 2024-09-17T15:54:13.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/2497971968 shutdown_connections 2024-09-17T15:54:13.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.778+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/2497971968 >> 172.21.15.64:0/2497971968 conn(0x7f6a2806db30 msgr2=0x7f6a28133e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:13.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.779+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/2497971968 shutdown_connections 2024-09-17T15:54:13.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.779+0000 7f6a2fe2a640 1 -- 172.21.15.64:0/2497971968 wait complete. 2024-09-17T15:54:13.820 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-17T15:54:13.820 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:54:13.821 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:54:13.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.919+0000 7f270879b640 10 monclient: get_monmap_and_config 2024-09-17T15:54:13.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.919+0000 7f270879b640 10 monclient: build_initial_monmap 2024-09-17T15:54:13.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.919+0000 7f270879b640 10 monclient: monmap: 2024-09-17T15:54:13.922 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:13.922 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.922 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:13.920807+0000 2024-09-17T15:54:13.922 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:13.920807+0000 2024-09-17T15:54:13.922 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:13.922 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.922 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:13.922 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:13.922 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:13.922 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:13.923 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:13.923 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:13.923 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:13.923 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:13.923 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:13.923 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.920+0000 7f270879b640 1 Processor -- start 2024-09-17T15:54:13.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.920+0000 7f270879b640 1 -- start start 2024-09-17T15:54:13.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.920+0000 7f270879b640 10 monclient: init 2024-09-17T15:54:13.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f270879b640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f270879b640 10 monclient: _add_conns ranks=[2,7,4,8,1,5,6,3,0] 2024-09-17T15:54:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f270879b640 10 monclient(hunting): picked mon.noname-b con 0x7f27000a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f270879b640 10 monclient(hunting): picked mon.noname-i con 0x7f2700135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f270879b640 10 monclient(hunting): picked mon.noname-c con 0x7f2700135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f270879b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270009d6f0 con 0x7f27000a5520 2024-09-17T15:54:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f270879b640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270009d9d0 con 0x7f2700135f70 2024-09-17T15:54:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f270879b640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270009dcb0 con 0x7f2700135b50 2024-09-17T15:54:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f270879b640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f270879b640 10 monclient(hunting): authenticate will time out at 1251.596401s 2024-09-17T15:54:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2706d38640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f2700135f70 0x7f27001415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:48244/0 (socket says 172.21.15.64:48244) 2024-09-17T15:54:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2706d38640 1 -- 172.21.15.64:0/2768812246 learned_addr learned my addr 172.21.15.64:0/2768812246 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2706537640 1 --1- 172.21.15.64:0/2768812246 >> v1:172.21.15.64:6790/0 conn(0x7f27000a5520 0x7f27000a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45662/0 (socket says 172.21.15.64:45662) 2024-09-17T15:54:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 1 -- 172.21.15.64:0/2768812246 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2771221726 0 0) 0x7f270009d6f0 con 0x7f27000a5520 2024-09-17T15:54:13.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 10 monclient(hunting): my global_id is 125264 2024-09-17T15:54:13.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 1 -- 172.21.15.64:0/2768812246 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26ec003520 con 0x7f27000a5520 2024-09-17T15:54:13.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 1 -- 172.21.15.64:0/2768812246 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3781968271 0 0) 0x7f270009d9d0 con 0x7f2700135f70 2024-09-17T15:54:13.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 10 monclient(hunting): my global_id is 154687 2024-09-17T15:54:13.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 1 -- 172.21.15.64:0/2768812246 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f270009d6f0 con 0x7f2700135f70 2024-09-17T15:54:13.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 1 -- 172.21.15.64:0/2768812246 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1871521301 0 0) 0x7f270009dcb0 con 0x7f2700135b50 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 10 monclient(hunting): my global_id is 174166 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.921+0000 7f2705535640 1 -- 172.21.15.64:0/2768812246 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f270009d9d0 con 0x7f2700135b50 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 1 -- 172.21.15.64:0/2768812246 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 717476854 0 0) 0x7f270009d6f0 con 0x7f2700135f70 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 1 -- 172.21.15.64:0/2768812246 >> v1:172.21.15.100:6792/0 conn(0x7f2700135b50 legacy=0x7f270013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 1 -- 172.21.15.64:0/2768812246 >> v1:172.21.15.64:6790/0 conn(0x7f27000a5520 legacy=0x7f27000a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 10 monclient: _finish_hunting 0 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 1 monclient: found mon.noname-c 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 1 -- 172.21.15.64:0/2768812246 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270009ccd0 con 0x7f2700135f70 2024-09-17T15:54:13.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 10 monclient: _finish_auth 0 2024-09-17T15:54:13.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 10 monclient: _check_auth_tickets 2024-09-17T15:54:13.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:13.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 1 -- 172.21.15.64:0/2768812246 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f26f8001b90 con 0x7f2700135f70 2024-09-17T15:54:13.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f270879b640 5 monclient: authenticate success, global_id 154687 2024-09-17T15:54:13.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f270879b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:13.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:13.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 10 monclient: dump: 2024-09-17T15:54:13.934 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.934 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.934 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.934 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f270879b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 1 -- 172.21.15.64:0/2768812246 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26f8001d30 con 0x7f2700135f70 2024-09-17T15:54:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f270879b640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 1 -- 172.21.15.64:0/2768812246 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f26f8004f60 con 0x7f2700135f70 2024-09-17T15:54:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f270879b640 4 set_mon_vals no callback set 2024-09-17T15:54:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f2705535640 10 monclient: dump: 2024-09-17T15:54:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.937 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.937 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.937 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.937 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.937 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.937 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.937 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.937 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.937 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.937 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f270879b640 10 monclient: shutdown 2024-09-17T15:54:13.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f270879b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:13.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f270879b640 1 -- 172.21.15.64:0/2768812246 >> v1:172.21.15.64:6791/0 conn(0x7f2700135f70 legacy=0x7f27001415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f270879b640 1 -- 172.21.15.64:0/2768812246 shutdown_connections 2024-09-17T15:54:13.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f270879b640 1 -- 172.21.15.64:0/2768812246 >> 172.21.15.64:0/2768812246 conn(0x7f270006db30 msgr2=0x7f27000ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:13.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f270879b640 1 -- 172.21.15.64:0/2768812246 shutdown_connections 2024-09-17T15:54:13.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.922+0000 7f270879b640 1 -- 172.21.15.64:0/2768812246 wait complete. 2024-09-17T15:54:13.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.923+0000 7f270879b640 10 monclient: build_initial_monmap 2024-09-17T15:54:13.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.923+0000 7f270879b640 10 monclient: monmap: 2024-09-17T15:54:13.938 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:13.938 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:13.938 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:13.938 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:13.939 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:13.939 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:13.939 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:13.939 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:13.939 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:13.939 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:13.939 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:13.939 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:13.939 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:13.939 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:13.939 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:13.939 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.923+0000 7f270879b640 1 Processor -- start 2024-09-17T15:54:13.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.923+0000 7f270879b640 1 -- start start 2024-09-17T15:54:13.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.923+0000 7f270879b640 10 monclient: init 2024-09-17T15:54:13.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f270879b640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:13.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f270879b640 10 monclient: _add_conns ranks=[5,7,2,6,4,3,0,8,1] 2024-09-17T15:54:13.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f270879b640 10 monclient(hunting): picked mon.noname-h con 0x7f27000a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:13.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f270879b640 10 monclient(hunting): picked mon.noname-i con 0x7f2700135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:13.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f270879b640 10 monclient(hunting): picked mon.noname-b con 0x7f2700135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:13.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f270879b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2700134f50 con 0x7f2700135f70 2024-09-17T15:54:13.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f270879b640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27000a0690 con 0x7f27000a5520 2024-09-17T15:54:13.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f270879b640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270009f820 con 0x7f2700135b50 2024-09-17T15:54:13.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f270879b640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:13.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f270879b640 10 monclient(hunting): authenticate will time out at 1251.599368s 2024-09-17T15:54:13.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f2706d38640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f2700135f70 0x7f27001d62b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45678/0 (socket says 172.21.15.64:45678) 2024-09-17T15:54:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f2706d38640 1 -- 172.21.15.64:0/2558814866 learned_addr learned my addr 172.21.15.64:0/2558814866 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 932961216 0 0) 0x7f270009ccd0 con 0x7f2700135f70 2024-09-17T15:54:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): my global_id is 125273 2024-09-17T15:54:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26e4003550 con 0x7f2700135f70 2024-09-17T15:54:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1142032840 0 0) 0x7f270009f820 con 0x7f2700135b50 2024-09-17T15:54:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): my global_id is 174175 2024-09-17T15:54:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f270009ccd0 con 0x7f2700135b50 2024-09-17T15:54:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4046002436 0 0) 0x7f27000a0690 con 0x7f27000a5520 2024-09-17T15:54:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): my global_id is 174254 2024-09-17T15:54:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f270009f820 con 0x7f27000a5520 2024-09-17T15:54:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2865554100 0 0) 0x7f26e4003550 con 0x7f2700135f70 2024-09-17T15:54:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27000a0690 con 0x7f2700135f70 2024-09-17T15:54:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f26f8004b60 con 0x7f2700135f70 2024-09-17T15:54:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:54:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): dump: 2024-09-17T15:54:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.944 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.924+0000 7f26f6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 850005991 0 0) 0x7f27000a0690 con 0x7f2700135f70 2024-09-17T15:54:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 >> v1:172.21.15.100:6792/0 conn(0x7f2700135b50 legacy=0x7f27001d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 >> v1:172.21.15.100:6791/0 conn(0x7f27000a5520 legacy=0x7f27000a4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:54:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 1 monclient: found mon.b 2024-09-17T15:54:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270009fdf0 con 0x7f2700135f70 2024-09-17T15:54:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:54:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:54:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f270879b640 5 monclient: authenticate success, global_id 125273 2024-09-17T15:54:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f270879b640 10 monclient: _renew_subs 2024-09-17T15:54:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f270879b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f270879b640 1 -- 172.21.15.64:0/2558814866 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f27000a5200 con 0x7f2700135f70 2024-09-17T15:54:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f270879b640 10 monclient: _renew_subs 2024-09-17T15:54:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f270879b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f270879b640 1 -- 172.21.15.64:0/2558814866 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f270009f070 con 0x7f2700135f70 2024-09-17T15:54:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26f8004fb0 con 0x7f2700135f70 2024-09-17T15:54:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f26f80054a0 con 0x7f2700135f70 2024-09-17T15:54:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f2704d34640 4 set_mon_vals no callback set 2024-09-17T15:54:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 10 monclient: dump: 2024-09-17T15:54:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:13.947 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:13.947 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:13.947 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:13.947 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:13.947 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:13.947 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:13.947 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:13.947 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:13.947 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:13.948 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:13.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.925+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f26f8011d80 con 0x7f2700135f70 2024-09-17T15:54:13.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.926+0000 7f270879b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:13.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.926+0000 7f270879b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:13.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.926+0000 7f270879b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:13.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.926+0000 7f270879b640 1 -- 172.21.15.64:0/2558814866 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26e8005080 con 0x7f2700135f70 2024-09-17T15:54:13.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.926+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f26f804edb0 con 0x7f2700135f70 2024-09-17T15:54:13.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.926+0000 7f2706537640 10 monclient: get_auth_request con 0x7f26e4040620 auth_method 0 2024-09-17T15:54:13.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.931+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f2700134f50 con 0x7f2700135f70 2024-09-17T15:54:13.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.931+0000 7f26f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:13.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:13.931+0000 7f26f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:14.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.187+0000 7f270879b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:54:14.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.187+0000 7f270879b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:54:14.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.187+0000 7f270879b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:14.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.187+0000 7f270879b640 1 -- 172.21.15.64:0/2558814866 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f26e8005b80 con 0x7f2700135f70 2024-09-17T15:54:14.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.189+0000 7f26f6ffd640 1 -- 172.21.15.64:0/2558814866 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f26f8017ac0 con 0x7f2700135f70 2024-09-17T15:54:14.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.189+0000 7f26f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:54:14.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.189+0000 7f26f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:14.191 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.191 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=10,kv=11,logm=51} crc {auth=1997226505,config=2586702516,health=1406995703,kv=4080546736,logm=1400213159}) 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=1997226505,config=2586702516,health=1486788974,kv=1246312931,logm=2090650910}) 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=10,kv=11,logm=51} crc {auth=1997226505,config=2586702516,health=1406995703,kv=4080546736,logm=1400213159}) 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=1,health=10,kv=12,logm=52} crc {auth=1324493078,config=1747058506,health=532415776,kv=1246312931,logm=2552192928}) 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=10,kv=11,logm=51} crc {auth=1997226505,config=2586702516,health=1406995703,kv=4080546736,logm=1400213159}) 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=3611565259,config=2586702516,health=3403977658,kv=1624202041,logm=49122556}) 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=10,kv=11,logm=51} crc {auth=1997226505,config=2586702516,health=1406995703,kv=4080546736,logm=1400213159}) 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=25,config=2,health=11,kv=11,logm=51} crc {auth=426569709,config=2586702516,health=4253015238,kv=4080546736,logm=3306669500}) 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=10,kv=11,logm=51} crc {auth=1997226505,config=2586702516,health=1406995703,kv=4080546736,logm=1400213159}) 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=26,config=2,health=12,kv=11,logm=49} crc {auth=1997226505,config=2586702516,health=1149746517,kv=403687235,logm=2358390080}) 2024-09-17T15:54:14.192 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.193 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=10,kv=11,logm=51} crc {auth=1997226505,config=2586702516,health=1406995703,kv=4080546736,logm=1400213159}) 2024-09-17T15:54:14.193 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=25,config=2,health=12,kv=12,logm=49} crc {auth=3733804451,config=2586702516,health=1149746517,kv=1246312931,logm=459440092}) 2024-09-17T15:54:14.193 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.193 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=10,kv=11,logm=51} crc {auth=1997226505,config=2586702516,health=1406995703,kv=4080546736,logm=1400213159}) 2024-09-17T15:54:14.193 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=25,config=2,health=12,kv=12,logm=49} crc {auth=3162447601,config=2586702516,health=1149746517,kv=1246312931,logm=3086906581}) 2024-09-17T15:54:14.193 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.193 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=10,kv=11,logm=51} crc {auth=1997226505,config=2586702516,health=1406995703,kv=4080546736,logm=1400213159}) 2024-09-17T15:54:14.193 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.190+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=25,config=2,health=12,kv=12,logm=49} crc {auth=3139801257,config=2586702516,health=3991619482,kv=1450974426,logm=2496222483}) 2024-09-17T15:54:14.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.191+0000 7f270879b640 1 -- 172.21.15.64:0/2558814866 >> v1:172.21.15.100:6812/4210963484 conn(0x7f26e4040620 legacy=0x7f26e4042ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:14.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.191+0000 7f270879b640 10 monclient: shutdown 2024-09-17T15:54:14.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.191+0000 7f270879b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:14.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.191+0000 7f270879b640 1 -- 172.21.15.64:0/2558814866 >> v1:172.21.15.64:6790/0 conn(0x7f2700135f70 legacy=0x7f27001d62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:14.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.191+0000 7f270879b640 1 -- 172.21.15.64:0/2558814866 shutdown_connections 2024-09-17T15:54:14.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.191+0000 7f270879b640 1 -- 172.21.15.64:0/2558814866 >> 172.21.15.64:0/2558814866 conn(0x7f270006db30 msgr2=0x7f270012c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:14.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.192+0000 7f270879b640 1 -- 172.21.15.64:0/2558814866 shutdown_connections 2024-09-17T15:54:14.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:14.192+0000 7f270879b640 1 -- 172.21.15.64:0/2558814866 wait complete. 2024-09-17T15:54:14.213 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:54:14.241 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.242 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408844951}) 2024-09-17T15:54:14.242 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2109193275}) 2024-09-17T15:54:14.242 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.242 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408844951}) 2024-09-17T15:54:14.242 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4020219180}) 2024-09-17T15:54:14.242 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.242 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408844951}) 2024-09-17T15:54:14.242 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3728806338}) 2024-09-17T15:54:14.242 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.242 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408844951}) 2024-09-17T15:54:14.242 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=843632398}) 2024-09-17T15:54:14.243 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.243 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408844951}) 2024-09-17T15:54:14.243 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=902423239}) 2024-09-17T15:54:14.243 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.243 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408844951}) 2024-09-17T15:54:14.243 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1990930035}) 2024-09-17T15:54:14.243 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.243 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408844951}) 2024-09-17T15:54:14.243 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2986512868}) 2024-09-17T15:54:14.243 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.243 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408844951}) 2024-09-17T15:54:14.243 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.191+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3071396208}) 2024-09-17T15:54:14.243 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.244 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2985458016}) 2024-09-17T15:54:14.244 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1592377843}) 2024-09-17T15:54:14.244 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.244 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2985458016}) 2024-09-17T15:54:14.244 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4259401593}) 2024-09-17T15:54:14.244 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.244 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2985458016}) 2024-09-17T15:54:14.244 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3609795329}) 2024-09-17T15:54:14.244 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.244 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2985458016}) 2024-09-17T15:54:14.244 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1427315045}) 2024-09-17T15:54:14.244 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.245 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2985458016}) 2024-09-17T15:54:14.245 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=107822739}) 2024-09-17T15:54:14.245 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.245 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2985458016}) 2024-09-17T15:54:14.245 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4121164295}) 2024-09-17T15:54:14.245 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.245 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2985458016}) 2024-09-17T15:54:14.245 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=715341941}) 2024-09-17T15:54:14.245 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.245 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2985458016}) 2024-09-17T15:54:14.245 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.193+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2752465099}) 2024-09-17T15:54:14.245 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.194+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.245 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.194+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474421919}) 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.194+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1527679995}) 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474421919}) 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=819588547}) 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474421919}) 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2380658182}) 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474421919}) 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3227324266}) 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474421919}) 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2478226109}) 2024-09-17T15:54:14.246 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.247 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474421919}) 2024-09-17T15:54:14.247 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=810560779}) 2024-09-17T15:54:14.247 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.247 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474421919}) 2024-09-17T15:54:14.247 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3201535447}) 2024-09-17T15:54:14.247 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.247 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474421919}) 2024-09-17T15:54:14.247 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.195+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=488795926}) 2024-09-17T15:54:14.247 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.247 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=430831565}) 2024-09-17T15:54:14.247 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=862704724}) 2024-09-17T15:54:14.247 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.247 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=430831565}) 2024-09-17T15:54:14.248 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310125800}) 2024-09-17T15:54:14.248 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.248 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=430831565}) 2024-09-17T15:54:14.248 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3157957546}) 2024-09-17T15:54:14.248 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.248 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=430831565}) 2024-09-17T15:54:14.248 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4173093281}) 2024-09-17T15:54:14.248 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.248 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=430831565}) 2024-09-17T15:54:14.248 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3601963364}) 2024-09-17T15:54:14.248 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.248 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=430831565}) 2024-09-17T15:54:14.248 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=585997429}) 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=430831565}) 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2438842077}) 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=430831565}) 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.196+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3360623766}) 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1785872862}) 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=816105932}) 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1785872862}) 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1554147013}) 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.249 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1785872862}) 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1367023211}) 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1785872862}) 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3799502865}) 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1785872862}) 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1287857432}) 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.197+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.198+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1785872862}) 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.198+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=385718852}) 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.198+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.198+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1785872862}) 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.198+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2474671649}) 2024-09-17T15:54:14.250 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.198+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.198+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1785872862}) 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.198+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=137312090}) 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057115862}) 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3488539804}) 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057115862}) 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4146667044}) 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057115862}) 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1315123623}) 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057115862}) 2024-09-17T15:54:14.251 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1215037010}) 2024-09-17T15:54:14.252 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.252 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057115862}) 2024-09-17T15:54:14.252 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=106910337}) 2024-09-17T15:54:14.252 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.252 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057115862}) 2024-09-17T15:54:14.252 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1012906701}) 2024-09-17T15:54:14.252 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.252 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057115862}) 2024-09-17T15:54:14.252 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=293365228}) 2024-09-17T15:54:14.252 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.252 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057115862}) 2024-09-17T15:54:14.252 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.199+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=340399538}) 2024-09-17T15:54:14.252 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1033325289}) 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954947153}) 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1033325289}) 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3713041817}) 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1033325289}) 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1271794832}) 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1033325289}) 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=712415554}) 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1033325289}) 2024-09-17T15:54:14.253 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2470590382}) 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1033325289}) 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2872261715}) 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1033325289}) 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3550958746}) 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1033325289}) 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.200+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708084493}) 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105922328}) 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1254135605}) 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.254 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105922328}) 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1947701563}) 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105922328}) 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3257652946}) 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105922328}) 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3188070039}) 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105922328}) 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3920638647}) 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105922328}) 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=441070098}) 2024-09-17T15:54:14.255 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.256 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105922328}) 2024-09-17T15:54:14.256 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2932094717}) 2024-09-17T15:54:14.256 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.256 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.201+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105922328}) 2024-09-17T15:54:14.256 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.202+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1322437246}) 2024-09-17T15:54:14.256 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.256 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1127705922}) 2024-09-17T15:54:14.256 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3535028950}) 2024-09-17T15:54:14.256 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.256 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1127705922}) 2024-09-17T15:54:14.256 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3595848410}) 2024-09-17T15:54:14.256 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.256 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1127705922}) 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1299210382}) 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1127705922}) 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1928240752}) 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1127705922}) 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3766937853}) 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1127705922}) 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4122011370}) 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1127705922}) 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=906013008}) 2024-09-17T15:54:14.257 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1127705922}) 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.203+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=334272668}) 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855735425}) 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3734440894}) 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855735425}) 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642713531}) 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855735425}) 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=813281922}) 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855735425}) 2024-09-17T15:54:14.258 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=860503722}) 2024-09-17T15:54:14.259 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.259 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855735425}) 2024-09-17T15:54:14.259 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3846996699}) 2024-09-17T15:54:14.259 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.259 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855735425}) 2024-09-17T15:54:14.259 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3924423742}) 2024-09-17T15:54:14.259 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.259 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855735425}) 2024-09-17T15:54:14.259 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3164861490}) 2024-09-17T15:54:14.259 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.259 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855735425}) 2024-09-17T15:54:14.259 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.204+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3328009098}) 2024-09-17T15:54:14.259 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020788134}) 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3867113549}) 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020788134}) 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516791080}) 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020788134}) 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=56119382}) 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020788134}) 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1069028369}) 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020788134}) 2024-09-17T15:54:14.260 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3312132239}) 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020788134}) 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1393643383}) 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020788134}) 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1253692846}) 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020788134}) 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.205+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1763648074}) 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.206+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.206+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3218967600}) 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.206+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2281723616}) 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.206+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.261 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.206+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3218967600}) 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.206+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4055186417}) 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.206+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3218967600}) 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2900417386}) 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3218967600}) 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3828582732}) 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3218967600}) 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1215285632}) 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3218967600}) 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3103275199}) 2024-09-17T15:54:14.262 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3218967600}) 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2148809044}) 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3218967600}) 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.207+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4208196704}) 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940312993}) 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3298624637}) 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940312993}) 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1553230750}) 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940312993}) 2024-09-17T15:54:14.263 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=947428387}) 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940312993}) 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3875243189}) 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940312993}) 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2403300517}) 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940312993}) 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3051783148}) 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940312993}) 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3645743696}) 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.264 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940312993}) 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.208+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3376105864}) 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512498222}) 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2458900452}) 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512498222}) 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1254765889}) 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512498222}) 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3973168532}) 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512498222}) 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1418589795}) 2024-09-17T15:54:14.265 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512498222}) 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=174682420}) 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512498222}) 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=523111910}) 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512498222}) 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3922910093}) 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512498222}) 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.209+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2808149498}) 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.210+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.210+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713048826}) 2024-09-17T15:54:14.266 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2853376845}) 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713048826}) 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4257186922}) 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713048826}) 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2316602483}) 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713048826}) 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3143548968}) 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713048826}) 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1500234455}) 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.267 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713048826}) 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1294821530}) 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713048826}) 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1461248337}) 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713048826}) 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.211+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3717717013}) 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2860554089}) 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665025288}) 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2860554089}) 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2678051700}) 2024-09-17T15:54:14.268 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2860554089}) 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2541374340}) 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2860554089}) 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=467109604}) 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2860554089}) 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3798410457}) 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2860554089}) 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3238797328}) 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2860554089}) 2024-09-17T15:54:14.269 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3961886048}) 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2860554089}) 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.212+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=566210617}) 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495607816}) 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644441915}) 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495607816}) 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3993457006}) 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495607816}) 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=270373926}) 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.270 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495607816}) 2024-09-17T15:54:14.271 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3999416228}) 2024-09-17T15:54:14.271 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.271 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495607816}) 2024-09-17T15:54:14.271 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3894084461}) 2024-09-17T15:54:14.271 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.271 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495607816}) 2024-09-17T15:54:14.271 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3181004880}) 2024-09-17T15:54:14.271 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.271 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495607816}) 2024-09-17T15:54:14.271 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1647802397}) 2024-09-17T15:54:14.271 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.271 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495607816}) 2024-09-17T15:54:14.271 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.213+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=224136046}) 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704046211}) 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252093149}) 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704046211}) 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=698121658}) 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704046211}) 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1032478164}) 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704046211}) 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1065225832}) 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704046211}) 2024-09-17T15:54:14.272 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=648046415}) 2024-09-17T15:54:14.273 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.273 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704046211}) 2024-09-17T15:54:14.273 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2272922771}) 2024-09-17T15:54:14.273 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.273 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704046211}) 2024-09-17T15:54:14.273 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1263257818}) 2024-09-17T15:54:14.273 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.273 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704046211}) 2024-09-17T15:54:14.273 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.215+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3938695034}) 2024-09-17T15:54:14.273 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.273 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947195687}) 2024-09-17T15:54:14.273 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599500123}) 2024-09-17T15:54:14.273 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947195687}) 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1966133637}) 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947195687}) 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2908743149}) 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947195687}) 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4067512993}) 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947195687}) 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2838126186}) 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947195687}) 2024-09-17T15:54:14.274 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2462971160}) 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947195687}) 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=967918747}) 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947195687}) 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.216+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3193640293}) 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4182545394}) 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384882967}) 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4182545394}) 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1751554237}) 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.275 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4182545394}) 2024-09-17T15:54:14.276 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1936463088}) 2024-09-17T15:54:14.276 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.276 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4182545394}) 2024-09-17T15:54:14.276 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=855479465}) 2024-09-17T15:54:14.276 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.276 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.217+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4182545394}) 2024-09-17T15:54:14.276 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.218+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4160471917}) 2024-09-17T15:54:14.276 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.218+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.276 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.218+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4182545394}) 2024-09-17T15:54:14.276 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.218+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=382567170}) 2024-09-17T15:54:14.276 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.218+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.276 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.218+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4182545394}) 2024-09-17T15:54:14.276 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.218+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3844558536}) 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.218+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.218+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4182545394}) 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.218+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1138119871}) 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147826230}) 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1252205716}) 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147826230}) 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=583972045}) 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147826230}) 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1143936313}) 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.277 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147826230}) 2024-09-17T15:54:14.278 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=71612261}) 2024-09-17T15:54:14.278 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.278 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147826230}) 2024-09-17T15:54:14.278 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2794267450}) 2024-09-17T15:54:14.278 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.278 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147826230}) 2024-09-17T15:54:14.278 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2681031838}) 2024-09-17T15:54:14.278 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.278 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147826230}) 2024-09-17T15:54:14.278 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=197218288}) 2024-09-17T15:54:14.278 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.278 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147826230}) 2024-09-17T15:54:14.278 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.219+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1430296332}) 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583341435}) 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1567171795}) 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583341435}) 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2758551698}) 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583341435}) 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3971545167}) 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583341435}) 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4256020794}) 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.279 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583341435}) 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.220+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2023451939}) 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.221+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.221+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583341435}) 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.221+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2414719277}) 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.221+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.221+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583341435}) 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.221+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3655393440}) 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.221+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.221+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583341435}) 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.221+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3255913276}) 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029158177}) 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222150531}) 2024-09-17T15:54:14.280 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.281 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029158177}) 2024-09-17T15:54:14.281 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1431200500}) 2024-09-17T15:54:14.281 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.281 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029158177}) 2024-09-17T15:54:14.281 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3685213336}) 2024-09-17T15:54:14.281 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.281 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029158177}) 2024-09-17T15:54:14.281 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2921156837}) 2024-09-17T15:54:14.281 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.281 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029158177}) 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4277227086}) 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029158177}) 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3712320768}) 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029158177}) 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2990962746}) 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029158177}) 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.222+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2358625226}) 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766652767}) 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028013467}) 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.282 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766652767}) 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1646048112}) 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766652767}) 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3734812625}) 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766652767}) 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=473550486}) 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766652767}) 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1002492580}) 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766652767}) 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2021734821}) 2024-09-17T15:54:14.283 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.284 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766652767}) 2024-09-17T15:54:14.284 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2218968334}) 2024-09-17T15:54:14.284 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.284 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766652767}) 2024-09-17T15:54:14.284 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.223+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4091141418}) 2024-09-17T15:54:14.284 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.224+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.284 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.224+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614411446}) 2024-09-17T15:54:14.284 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.224+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952968934}) 2024-09-17T15:54:14.284 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.224+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.284 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.224+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614411446}) 2024-09-17T15:54:14.284 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.224+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1781258139}) 2024-09-17T15:54:14.284 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.284 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614411446}) 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1548440798}) 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614411446}) 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1448046723}) 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614411446}) 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2175803554}) 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614411446}) 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4246099297}) 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614411446}) 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2944507537}) 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.285 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614411446}) 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.225+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=69343127}) 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3391850068}) 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3173547428}) 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3391850068}) 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1760167406}) 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3391850068}) 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3082543707}) 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3391850068}) 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2663169959}) 2024-09-17T15:54:14.286 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.287 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3391850068}) 2024-09-17T15:54:14.287 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1326995613}) 2024-09-17T15:54:14.287 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.287 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3391850068}) 2024-09-17T15:54:14.287 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2299022060}) 2024-09-17T15:54:14.287 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.287 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3391850068}) 2024-09-17T15:54:14.287 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3616736021}) 2024-09-17T15:54:14.287 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.287 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3391850068}) 2024-09-17T15:54:14.287 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.226+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1196708590}) 2024-09-17T15:54:14.287 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.287 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2216006391}) 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175959759}) 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2216006391}) 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=282196057}) 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2216006391}) 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1497859526}) 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2216006391}) 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1060927862}) 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2216006391}) 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2957789848}) 2024-09-17T15:54:14.288 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.289 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2216006391}) 2024-09-17T15:54:14.289 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1959683998}) 2024-09-17T15:54:14.289 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.289 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2216006391}) 2024-09-17T15:54:14.289 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1581997687}) 2024-09-17T15:54:14.289 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.289 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2216006391}) 2024-09-17T15:54:14.289 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.227+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3718248158}) 2024-09-17T15:54:14.289 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.228+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.289 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.228+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650149577}) 2024-09-17T15:54:14.289 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.228+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3690254869}) 2024-09-17T15:54:14.289 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.228+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.289 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.228+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650149577}) 2024-09-17T15:54:14.290 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.228+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871213477}) 2024-09-17T15:54:14.290 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.228+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.290 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.228+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650149577}) 2024-09-17T15:54:14.290 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.228+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1625290364}) 2024-09-17T15:54:14.290 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.228+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.290 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.228+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650149577}) 2024-09-17T15:54:14.290 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.228+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2239894654}) 2024-09-17T15:54:14.290 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.228+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.290 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.229+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650149577}) 2024-09-17T15:54:14.290 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.229+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2708385647}) 2024-09-17T15:54:14.290 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.229+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.290 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.229+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650149577}) 2024-09-17T15:54:14.290 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.229+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=824193238}) 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.229+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.229+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650149577}) 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.229+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1322593409}) 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.229+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.229+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650149577}) 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.229+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3407586887}) 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533164811}) 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=641920095}) 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533164811}) 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3691595299}) 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.291 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533164811}) 2024-09-17T15:54:14.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4144855807}) 2024-09-17T15:54:14.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533164811}) 2024-09-17T15:54:14.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=497165215}) 2024-09-17T15:54:14.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533164811}) 2024-09-17T15:54:14.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=157658159}) 2024-09-17T15:54:14.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533164811}) 2024-09-17T15:54:14.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2147111362}) 2024-09-17T15:54:14.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.292 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533164811}) 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4072784917}) 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533164811}) 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.230+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1064716743}) 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=3677871192,mdsmap=1462844619,mgr=745432537,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=145298935,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=1539608060}) 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=3,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=42} crc {logm=3677871192,mdsmap=3196345284,mgr=618461289,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=3921033350,mon_config_key=1328119858,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=4051620427}) 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=3677871192,mdsmap=1462844619,mgr=745432537,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=145298935,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=1539608060}) 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=3,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=45} crc {logm=3677871192,mdsmap=1462844619,mgr=4276097997,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=4241412192,mon_config_key=825509050,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=1782029848}) 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=3677871192,mdsmap=1462844619,mgr=745432537,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=145298935,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=1539608060}) 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=3677871192,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=3921033350,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=179057771,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=958376066}) 2024-09-17T15:54:14.293 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=3677871192,mdsmap=1462844619,mgr=745432537,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=145298935,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=1539608060}) 2024-09-17T15:54:14.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=3677871192,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=4126298529,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=3837883758}) 2024-09-17T15:54:14.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=3677871192,mdsmap=1462844619,mgr=745432537,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=145298935,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=1539608060}) 2024-09-17T15:54:14.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=42} crc {logm=3677871192,mdsmap=1462844619,mgr=1860383721,mgr_command_descs=1957073259,mgrstat=3921033350,mon_config_key=602012498,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=2910713269}) 2024-09-17T15:54:14.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=3677871192,mdsmap=1462844619,mgr=745432537,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=145298935,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=1539608060}) 2024-09-17T15:54:14.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=3,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=42} crc {logm=3677871192,mdsmap=1462844619,mgr=4116046384,mgr_metadata=4041113850,mgrstat=1630911345,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752440,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=2330521784}) 2024-09-17T15:54:14.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=3677871192,mdsmap=1462844619,mgr=745432537,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=145298935,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=1539608060}) 2024-09-17T15:54:14.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=3,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=43} crc {logm=3677871192,mdsmap=3140514009,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=1652136593,mon_config_key=825509050,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=1410388174}) 2024-09-17T15:54:14.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.294 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=3677871192,mdsmap=1462844619,mgr=745432537,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=145298935,mon_config_key=4501194,monmap=1573330833,nvmeofgw=598337686,osd_metadata=4100752439,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=1539608060}) 2024-09-17T15:54:14.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.232+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=42} crc {logm=3677871192,mdsmap=1462844619,mgr=4151545133,mgr_command_descs=1957073259,mgr_metadata=4041113850,mgrstat=92324083,mon_config_key=3407507196,monmap=1573330833,nvmeofgw=598337686,osd_metadata=2749459017,osd_pg_creating=3933572464,osd_snap=723727393,osdmap=3960309564}) 2024-09-17T15:54:14.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=3919369769}) 2024-09-17T15:54:14.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=32} crc {osdmap=649473661}) 2024-09-17T15:54:14.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=3919369769}) 2024-09-17T15:54:14.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=32} crc {osdmap=3521122447}) 2024-09-17T15:54:14.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=3919369769}) 2024-09-17T15:54:14.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=34} crc {osdmap=3555981429}) 2024-09-17T15:54:14.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=3919369769}) 2024-09-17T15:54:14.295 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=33} crc {osdmap=249412633}) 2024-09-17T15:54:14.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=3919369769}) 2024-09-17T15:54:14.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=32} crc {osdmap=2011170968}) 2024-09-17T15:54:14.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=3919369769}) 2024-09-17T15:54:14.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=33} crc {osdmap=1747928666}) 2024-09-17T15:54:14.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=3919369769}) 2024-09-17T15:54:14.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=33} crc {osdmap=3592895512}) 2024-09-17T15:54:14.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:54:14.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=3919369769}) 2024-09-17T15:54:14.296 INFO:tasks.ceph.mon.f.smithi100.stderr:2024-09-17T15:54:14.233+0000 7fde594dc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=32} crc {osdmap=1302714624}) 2024-09-17T15:54:15.214 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-17T15:54:15.215 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:54:15.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.316+0000 7fd9ee344640 10 monclient: get_monmap_and_config 2024-09-17T15:54:15.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.316+0000 7fd9ee344640 10 monclient: build_initial_monmap 2024-09-17T15:54:15.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.316+0000 7fd9ee344640 10 monclient: monmap: 2024-09-17T15:54:15.319 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:15.319 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:15.319 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:15.317673+0000 2024-09-17T15:54:15.319 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:15.317673+0000 2024-09-17T15:54:15.319 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:15.320 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:15.320 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:15.320 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:15.320 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:15.320 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:15.320 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:15.320 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:15.320 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:15.321 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:15.321 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:15.321 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.317+0000 7fd9ee344640 1 Processor -- start 2024-09-17T15:54:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.317+0000 7fd9ee344640 1 -- start start 2024-09-17T15:54:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.317+0000 7fd9ee344640 10 monclient: init 2024-09-17T15:54:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ee344640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ee344640 10 monclient: _add_conns ranks=[4,5,2,7,8,6,0,3,1] 2024-09-17T15:54:15.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ee344640 10 monclient(hunting): picked mon.noname-c con 0x7fd9e80a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:15.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ee344640 10 monclient(hunting): picked mon.noname-h con 0x7fd9e8135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:15.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ee344640 10 monclient(hunting): picked mon.noname-b con 0x7fd9e8135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:15.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ee344640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9e809d6f0 con 0x7fd9e8135f70 2024-09-17T15:54:15.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ee344640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9e809d9d0 con 0x7fd9e80a5520 2024-09-17T15:54:15.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ee344640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9e809dcb0 con 0x7fd9e8135b50 2024-09-17T15:54:15.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ee344640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:15.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ee344640 10 monclient(hunting): authenticate will time out at 1252.993218s 2024-09-17T15:54:15.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e7fff640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fd9e80a5520 0x7fd9e80a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:59832/0 (socket says 172.21.15.64:59832) 2024-09-17T15:54:15.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ec8e1640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fd9e8135f70 0x7fd9e81415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53600/0 (socket says 172.21.15.64:53600) 2024-09-17T15:54:15.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e7fff640 1 -- 172.21.15.64:0/1497695327 learned_addr learned my addr 172.21.15.64:0/1497695327 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:15.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 1 -- 172.21.15.64:0/1497695327 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1453041280 0 0) 0x7fd9e809d6f0 con 0x7fd9e8135f70 2024-09-17T15:54:15.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:15.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:15.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:15.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient(hunting): my global_id is 125282 2024-09-17T15:54:15.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 1 -- 172.21.15.64:0/1497695327 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9d0003520 con 0x7fd9e8135f70 2024-09-17T15:54:15.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 1 -- 172.21.15.64:0/1497695327 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2456091623 0 0) 0x7fd9e809d9d0 con 0x7fd9e80a5520 2024-09-17T15:54:15.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:15.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:15.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:15.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient(hunting): my global_id is 154696 2024-09-17T15:54:15.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 1 -- 172.21.15.64:0/1497695327 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9e809d6f0 con 0x7fd9e80a5520 2024-09-17T15:54:15.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 1 -- 172.21.15.64:0/1497695327 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 516816490 0 0) 0x7fd9d0003520 con 0x7fd9e8135f70 2024-09-17T15:54:15.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 1 -- 172.21.15.64:0/1497695327 >> v1:172.21.15.100:6791/0 conn(0x7fd9e8135b50 legacy=0x7fd9e813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 1 -- 172.21.15.64:0/1497695327 >> v1:172.21.15.64:6791/0 conn(0x7fd9e80a5520 legacy=0x7fd9e80a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:54:15.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 1 monclient: found mon.noname-b 2024-09-17T15:54:15.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:15.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:54:15.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 1 -- 172.21.15.64:0/1497695327 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e809ccd0 con 0x7fd9e8135f70 2024-09-17T15:54:15.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:54:15.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:54:15.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:15.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 1 -- 172.21.15.64:0/1497695327 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd9e0001bb0 con 0x7fd9e8135f70 2024-09-17T15:54:15.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ee344640 5 monclient: authenticate success, global_id 125282 2024-09-17T15:54:15.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ee344640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:15.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:15.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:54:15.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9e6ffd640 10 monclient: dump: 2024-09-17T15:54:15.344 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:15.344 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:15.344 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.345 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.345 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:15.345 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:15.345 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:15.345 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:15.345 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:15.345 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:15.345 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:15.345 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:15.345 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:15.345 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:15.345 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:15.346 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.318+0000 7fd9ee344640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9e6ffd640 1 -- 172.21.15.64:0/1497695327 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9e0001d50 con 0x7fd9e8135f70 2024-09-17T15:54:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9e6ffd640 1 -- 172.21.15.64:0/1497695327 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd9e0004ea0 con 0x7fd9e8135f70 2024-09-17T15:54:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9ee344640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9ee344640 4 set_mon_vals no callback set 2024-09-17T15:54:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9e6ffd640 10 monclient: dump: 2024-09-17T15:54:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:15.347 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.347 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.347 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:15.347 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:15.347 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:15.347 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:15.347 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:15.348 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:15.348 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:15.348 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:15.348 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:15.348 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:15.348 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:15.348 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9ee344640 10 monclient: shutdown 2024-09-17T15:54:15.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9ee344640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:15.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9ee344640 1 -- 172.21.15.64:0/1497695327 >> v1:172.21.15.64:6790/0 conn(0x7fd9e8135f70 legacy=0x7fd9e81415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9ee344640 1 -- 172.21.15.64:0/1497695327 shutdown_connections 2024-09-17T15:54:15.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9ee344640 1 -- 172.21.15.64:0/1497695327 >> 172.21.15.64:0/1497695327 conn(0x7fd9e806db30 msgr2=0x7fd9e80ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9ee344640 1 -- 172.21.15.64:0/1497695327 shutdown_connections 2024-09-17T15:54:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.319+0000 7fd9ee344640 1 -- 172.21.15.64:0/1497695327 wait complete. 2024-09-17T15:54:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.320+0000 7fd9ee344640 10 monclient: build_initial_monmap 2024-09-17T15:54:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.320+0000 7fd9ee344640 10 monclient: monmap: 2024-09-17T15:54:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:15.350 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:15.350 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:15.350 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:15.350 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:15.350 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:15.350 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:15.350 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.320+0000 7fd9ee344640 1 Processor -- start 2024-09-17T15:54:15.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.320+0000 7fd9ee344640 1 -- start start 2024-09-17T15:54:15.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.320+0000 7fd9ee344640 10 monclient: init 2024-09-17T15:54:15.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:15.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 10 monclient: _add_conns ranks=[8,5,6,0,1,7,3,2,4] 2024-09-17T15:54:15.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 10 monclient(hunting): picked mon.noname-e con 0x7fd9e80a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:15.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 10 monclient(hunting): picked mon.noname-h con 0x7fd9e8135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:15.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 10 monclient(hunting): picked mon.noname-d con 0x7fd9e8135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:15.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9e8135060 con 0x7fd9e8135b50 2024-09-17T15:54:15.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9e80a00c0 con 0x7fd9e8135f70 2024-09-17T15:54:15.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9e8132740 con 0x7fd9e80a5520 2024-09-17T15:54:15.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:15.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 10 monclient(hunting): authenticate will time out at 1252.996120s 2024-09-17T15:54:15.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e7fff640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fd9e80a5520 0x7fd9e8131920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51056/0 (socket says 172.21.15.64:51056) 2024-09-17T15:54:15.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e7fff640 1 -- 172.21.15.64:0/1806065288 learned_addr learned my addr 172.21.15.64:0/1806065288 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:15.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1126654503 0 0) 0x7fd9e80a00c0 con 0x7fd9e8135f70 2024-09-17T15:54:15.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:54:15.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:15.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:15.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): my global_id is 174201 2024-09-17T15:54:15.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9d4003520 con 0x7fd9e8135f70 2024-09-17T15:54:15.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3866257753 0 0) 0x7fd9e8132740 con 0x7fd9e80a5520 2024-09-17T15:54:15.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:54:15.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:15.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:15.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): my global_id is 174185 2024-09-17T15:54:15.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9e80a00c0 con 0x7fd9e80a5520 2024-09-17T15:54:15.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1737364596 0 0) 0x7fd9e80a00c0 con 0x7fd9e80a5520 2024-09-17T15:54:15.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9e8132740 con 0x7fd9e80a5520 2024-09-17T15:54:15.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd9d8001650 con 0x7fd9e80a5520 2024-09-17T15:54:15.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:15.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:54:15.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): dump: 2024-09-17T15:54:15.353 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:15.353 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:15.353 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.353 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.353 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:15.353 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3627080944 0 0) 0x7fd9d4003520 con 0x7fd9e8135f70 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9e80a00c0 con 0x7fd9e8135f70 2024-09-17T15:54:15.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd9e0001af0 con 0x7fd9e8135f70 2024-09-17T15:54:15.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:15.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:15.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): dump: 2024-09-17T15:54:15.355 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:15.355 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:15.355 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.355 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.355 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:15.355 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:15.355 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:15.355 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:15.355 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:15.355 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:15.356 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:15.356 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:15.356 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:15.356 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:15.356 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:15.356 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3614802908 0 0) 0x7fd9dc001230 con 0x7fd9e8135b50 2024-09-17T15:54:15.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): _negotiate 2024-09-17T15:54:15.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:15.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:15.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient(hunting): my global_id is 174272 2024-09-17T15:54:15.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9d4003520 con 0x7fd9e8135b50 2024-09-17T15:54:15.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 835421715 0 0) 0x7fd9e8132740 con 0x7fd9e80a5520 2024-09-17T15:54:15.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 >> v1:172.21.15.64:6792/0 conn(0x7fd9e8135f70 legacy=0x7fd9e81d5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 >> v1:172.21.15.100:6791/0 conn(0x7fd9e8135b50 legacy=0x7fd9e8132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient: _finish_hunting 0 2024-09-17T15:54:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 monclient: found mon.e 2024-09-17T15:54:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e8134d10 con 0x7fd9e80a5520 2024-09-17T15:54:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient: _finish_auth 0 2024-09-17T15:54:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:54:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 5 monclient: authenticate success, global_id 174185 2024-09-17T15:54:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 10 monclient: _renew_subs 2024-09-17T15:54:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.321+0000 7fd9ee344640 1 -- 172.21.15.64:0/1806065288 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd9e80a8e00 con 0x7fd9e80a5520 2024-09-17T15:54:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9ee344640 10 monclient: _renew_subs 2024-09-17T15:54:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9ee344640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:15.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9ee344640 1 -- 172.21.15.64:0/1806065288 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fd9e809e720 con 0x7fd9e80a5520 2024-09-17T15:54:15.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9d8004a90 con 0x7fd9e80a5520 2024-09-17T15:54:15.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9e4ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:15.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd9d8004f80 con 0x7fd9e80a5520 2024-09-17T15:54:15.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:15.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9e4ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:15.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9e67fc640 4 set_mon_vals no callback set 2024-09-17T15:54:15.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9e4ff9640 10 monclient: dump: 2024-09-17T15:54:15.359 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:15.359 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:15.359 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.359 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.359 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:15.360 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:15.360 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:15.360 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:15.360 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:15.360 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:15.360 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:15.360 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:15.360 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:15.360 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:15.360 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:15.360 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fd9d8011860 con 0x7fd9e80a5520 2024-09-17T15:54:15.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9ee344640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:15.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9ee344640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:15.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9ee344640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:15.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.322+0000 7fd9ee344640 1 -- 172.21.15.64:0/1806065288 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c8005080 con 0x7fd9e80a5520 2024-09-17T15:54:15.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.323+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7fd9d804eae0 con 0x7fd9e80a5520 2024-09-17T15:54:15.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.323+0000 7fd9e77fe640 10 monclient: get_auth_request con 0x7fd9d4040a50 auth_method 0 2024-09-17T15:54:15.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.327+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.8 v1:172.21.15.64:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fd9d8017910 con 0x7fd9e80a5520 2024-09-17T15:54:15.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.327+0000 7fd9e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:15.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.327+0000 7fd9e4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:15.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.585+0000 7fd9ee344640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:54:15.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.585+0000 7fd9ee344640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:54:15.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.585+0000 7fd9ee344640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:15.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.585+0000 7fd9ee344640 1 -- 172.21.15.64:0/1806065288 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd9c8005370 con 0x7fd9e80a5520 2024-09-17T15:54:15.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.585+0000 7fd9e4ff9640 1 -- 172.21.15.64:0/1806065288 <== mon.8 v1:172.21.15.64:6793/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 1495107919) 0x7fd9d8017210 con 0x7fd9e80a5520 2024-09-17T15:54:15.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.585+0000 7fd9e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:54:15.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.585+0000 7fd9e4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:15.588 INFO:teuthology.orchestra.run.smithi064.stdout:{"election_epoch":46,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","modified":"2024-09-17T15:45:15.994818Z","created":"2024-09-17T15:45:15.994818Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6791","nonce":0}]},"addr":"172.21.15.64:6791/0","public_addr":"172.21.15.64:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6791","nonce":0}]},"addr":"172.21.15.100:6791/0","public_addr":"172.21.15.100:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6792","nonce":0}]},"addr":"172.21.15.64:6792/0","public_addr":"172.21.15.64:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6792","nonce":0}]},"addr":"172.21.15.100:6792/0","public_addr":"172.21.15.100:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6793","nonce":0}]},"addr":"172.21.15.64:6793/0","public_addr":"172.21.15.64:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:54:15.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.587+0000 7fd9ee344640 1 -- 172.21.15.64:0/1806065288 >> v1:172.21.15.100:6812/4210963484 conn(0x7fd9d4040a50 legacy=0x7fd9d4042f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.587+0000 7fd9ee344640 10 monclient: shutdown 2024-09-17T15:54:15.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.587+0000 7fd9ee344640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:15.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.587+0000 7fd9ee344640 1 -- 172.21.15.64:0/1806065288 >> v1:172.21.15.64:6793/0 conn(0x7fd9e80a5520 legacy=0x7fd9e8131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.587+0000 7fd9ee344640 1 -- 172.21.15.64:0/1806065288 shutdown_connections 2024-09-17T15:54:15.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.587+0000 7fd9ee344640 1 -- 172.21.15.64:0/1806065288 >> 172.21.15.64:0/1806065288 conn(0x7fd9e806db30 msgr2=0x7fd9e81334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:15.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.587+0000 7fd9ee344640 1 -- 172.21.15.64:0/1806065288 shutdown_connections 2024-09-17T15:54:15.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.587+0000 7fd9ee344640 1 -- 172.21.15.64:0/1806065288 wait complete. 2024-09-17T15:54:15.600 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-17T15:54:15.600 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-17T15:54:15.669 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T15:54:15.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:54:15.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.768+0000 7fce47a73640 10 monclient: get_monmap_and_config 2024-09-17T15:54:15.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.768+0000 7fce47a73640 10 monclient: build_initial_monmap 2024-09-17T15:54:15.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.768+0000 7fce47a73640 10 monclient: monmap: 2024-09-17T15:54:15.770 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:15.771 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:15.771 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:15.769483+0000 2024-09-17T15:54:15.771 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:15.769483+0000 2024-09-17T15:54:15.771 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:15.771 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:15.771 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:15.771 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:15.771 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:15.771 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:15.771 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:15.772 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:15.772 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:15.772 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:15.772 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:15.772 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.769+0000 7fce47a73640 1 Processor -- start 2024-09-17T15:54:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.769+0000 7fce47a73640 1 -- start start 2024-09-17T15:54:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.769+0000 7fce47a73640 10 monclient: init 2024-09-17T15:54:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.769+0000 7fce47a73640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.769+0000 7fce47a73640 10 monclient: _add_conns ranks=[6,5,4,2,7,8,3,1,0] 2024-09-17T15:54:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.769+0000 7fce47a73640 10 monclient(hunting): picked mon.noname-d con 0x7fce400a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.769+0000 7fce47a73640 10 monclient(hunting): picked mon.noname-h con 0x7fce40135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce47a73640 10 monclient(hunting): picked mon.noname-c con 0x7fce40135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce47a73640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce4009d6f0 con 0x7fce40135f70 2024-09-17T15:54:15.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce47a73640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce4009d9d0 con 0x7fce40135b50 2024-09-17T15:54:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce47a73640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce4009dcb0 con 0x7fce400a5520 2024-09-17T15:54:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce47a73640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce47a73640 10 monclient(hunting): authenticate will time out at 1253.445075s 2024-09-17T15:54:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4580f640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fce400a5520 0x7fce400a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:54814/0 (socket says 172.21.15.64:54814) 2024-09-17T15:54:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4580f640 1 -- 172.21.15.64:0/2818596978 learned_addr learned my addr 172.21.15.64:0/2818596978 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 1 -- 172.21.15.64:0/2818596978 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1047729000 0 0) 0x7fce4009d6f0 con 0x7fce40135f70 2024-09-17T15:54:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient(hunting): _negotiate 2024-09-17T15:54:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient(hunting): my global_id is 154705 2024-09-17T15:54:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 1 -- 172.21.15.64:0/2818596978 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce2c003520 con 0x7fce40135f70 2024-09-17T15:54:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 1 -- 172.21.15.64:0/2818596978 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355567863 0 0) 0x7fce4009dcb0 con 0x7fce400a5520 2024-09-17T15:54:15.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient(hunting): _negotiate 2024-09-17T15:54:15.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:15.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:15.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient(hunting): my global_id is 174210 2024-09-17T15:54:15.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 1 -- 172.21.15.64:0/2818596978 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce4009d6f0 con 0x7fce400a5520 2024-09-17T15:54:15.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 1 -- 172.21.15.64:0/2818596978 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3163334656 0 0) 0x7fce2c003520 con 0x7fce40135f70 2024-09-17T15:54:15.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 1 -- 172.21.15.64:0/2818596978 >> v1:172.21.15.64:6792/0 conn(0x7fce400a5520 legacy=0x7fce400a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 1 -- 172.21.15.64:0/2818596978 >> v1:172.21.15.100:6791/0 conn(0x7fce40135b50 legacy=0x7fce4013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient: _finish_hunting 0 2024-09-17T15:54:15.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 1 monclient: found mon.noname-c 2024-09-17T15:54:15.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:15.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:54:15.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 1 -- 172.21.15.64:0/2818596978 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4009ccd0 con 0x7fce40135f70 2024-09-17T15:54:15.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient: _finish_auth 0 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient: _check_auth_tickets 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 1 -- 172.21.15.64:0/2818596978 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fce38001bb0 con 0x7fce40135f70 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce47a73640 5 monclient: authenticate success, global_id 154705 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce47a73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient: dump: 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:15.783 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:15.784 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:15.784 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:15.784 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:15.784 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:15.784 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:15.784 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:15.784 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:15.784 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:15.784 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:15.784 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce47a73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:15.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 1 -- 172.21.15.64:0/2818596978 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fce38001dd0 con 0x7fce40135f70 2024-09-17T15:54:15.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 1 -- 172.21.15.64:0/2818596978 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fce380020c0 con 0x7fce40135f70 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce47a73640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce47a73640 4 set_mon_vals no callback set 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.770+0000 7fce4480d640 10 monclient: dump: 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:15.785 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.771+0000 7fce47a73640 10 monclient: shutdown 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.771+0000 7fce47a73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.771+0000 7fce47a73640 1 -- 172.21.15.64:0/2818596978 >> v1:172.21.15.64:6791/0 conn(0x7fce40135f70 legacy=0x7fce401415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.771+0000 7fce47a73640 1 -- 172.21.15.64:0/2818596978 shutdown_connections 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.771+0000 7fce47a73640 1 -- 172.21.15.64:0/2818596978 >> 172.21.15.64:0/2818596978 conn(0x7fce4006db30 msgr2=0x7fce400ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:15.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.771+0000 7fce47a73640 1 -- 172.21.15.64:0/2818596978 shutdown_connections 2024-09-17T15:54:15.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.771+0000 7fce47a73640 1 -- 172.21.15.64:0/2818596978 wait complete. 2024-09-17T15:54:15.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.771+0000 7fce47a73640 10 monclient: build_initial_monmap 2024-09-17T15:54:15.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 10 monclient: monmap: 2024-09-17T15:54:15.787 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:15.787 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:15.787 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:15.787 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:15.787 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:15.787 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:15.787 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:15.787 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:15.788 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:15.788 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:15.788 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:15.788 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:15.788 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:15.788 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:15.788 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:15.788 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 1 Processor -- start 2024-09-17T15:54:15.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 1 -- start start 2024-09-17T15:54:15.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 10 monclient: init 2024-09-17T15:54:15.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:15.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 10 monclient: _add_conns ranks=[0,8,1,4,2,6,3,5,7] 2024-09-17T15:54:15.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 10 monclient(hunting): picked mon.noname-a con 0x7fce400a5520 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:15.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 10 monclient(hunting): picked mon.noname-e con 0x7fce40135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:15.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 10 monclient(hunting): picked mon.noname-f con 0x7fce40135f70 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:15.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce4013a290 con 0x7fce400a5520 2024-09-17T15:54:15.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce40135310 con 0x7fce40135f70 2024-09-17T15:54:15.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce40134f50 con 0x7fce40135b50 2024-09-17T15:54:15.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:15.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce47a73640 10 monclient(hunting): authenticate will time out at 1253.447960s 2024-09-17T15:54:15.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.772+0000 7fce4580f640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fce400a5520 0x7fce40136ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35308/0 (socket says 172.21.15.64:35308) 2024-09-17T15:54:15.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce4580f640 1 -- 172.21.15.64:0/785028007 learned_addr learned my addr 172.21.15.64:0/785028007 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:15.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce4500e640 1 --1- 172.21.15.64:0/785028007 >> v1:172.21.15.64:6793/0 conn(0x7fce40135b50 0x7fce401373f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51070/0 (socket says 172.21.15.64:51070) 2024-09-17T15:54:15.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2929052903 0 0) 0x7fce40134f50 con 0x7fce40135b50 2024-09-17T15:54:15.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:15.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:15.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:15.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): my global_id is 174194 2024-09-17T15:54:15.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce24003550 con 0x7fce40135b50 2024-09-17T15:54:15.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3630617229 0 0) 0x7fce3c001a10 con 0x7fce400a5520 2024-09-17T15:54:15.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:15.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:15.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:15.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): my global_id is 125145 2024-09-17T15:54:15.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce40134f50 con 0x7fce400a5520 2024-09-17T15:54:15.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 467112488 0 0) 0x7fce24003550 con 0x7fce40135b50 2024-09-17T15:54:15.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce3c001a10 con 0x7fce40135b50 2024-09-17T15:54:15.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fce34001b10 con 0x7fce40135b50 2024-09-17T15:54:15.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:15.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:54:15.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): dump: 2024-09-17T15:54:15.791 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:15.791 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:15.791 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.791 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.791 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:15.791 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:15.791 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:15.791 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:15.791 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:15.791 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:15.792 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:15.792 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:15.792 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:15.792 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:15.792 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:15.792 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:15.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1289123949 0 0) 0x7fce40134f50 con 0x7fce400a5520 2024-09-17T15:54:15.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce24003550 con 0x7fce400a5520 2024-09-17T15:54:15.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 796277708 0 0) 0x7fce40135310 con 0x7fce40135f70 2024-09-17T15:54:15.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:15.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): my global_id is 6949 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce40134f50 con 0x7fce40135f70 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fce3c0017c0 con 0x7fce400a5520 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient(hunting): dump: 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:15.793 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:15.794 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:15.794 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:15.794 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:15.794 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:15.794 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:15.794 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:15.794 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:15.794 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:15.794 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1342015900 0 0) 0x7fce3c001a10 con 0x7fce40135b50 2024-09-17T15:54:15.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 >> v1:172.21.15.100:6789/0 conn(0x7fce40135f70 legacy=0x7fce40137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 >> v1:172.21.15.64:6789/0 conn(0x7fce400a5520 legacy=0x7fce40136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient: _finish_hunting 0 2024-09-17T15:54:15.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 monclient: found mon.e 2024-09-17T15:54:15.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:15.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:15.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce401387d0 con 0x7fce40135b50 2024-09-17T15:54:15.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient: _finish_auth 0 2024-09-17T15:54:15.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient: _check_auth_tickets 2024-09-17T15:54:15.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:15.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce47a73640 5 monclient: authenticate success, global_id 174194 2024-09-17T15:54:15.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce47a73640 10 monclient: _renew_subs 2024-09-17T15:54:15.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce47a73640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:15.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce47a73640 1 -- 172.21.15.64:0/785028007 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fce40141830 con 0x7fce40135b50 2024-09-17T15:54:15.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce47a73640 10 monclient: _renew_subs 2024-09-17T15:54:15.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce47a73640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce47a73640 1 -- 172.21.15.64:0/785028007 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fce40139330 con 0x7fce40135b50 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fce340038c0 con 0x7fce40135b50 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fce34003e70 con 0x7fce40135b50 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce33fff640 4 set_mon_vals no callback set 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.773+0000 7fce327fc640 10 monclient: dump: 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:15.796 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.774+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fce34004090 con 0x7fce40135b50 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.774+0000 7fce47a73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.774+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7fce3404eb50 con 0x7fce40135b50 2024-09-17T15:54:15.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.774+0000 7fce47a73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:15.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.774+0000 7fce47a73640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:15.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.774+0000 7fce47a73640 1 -- 172.21.15.64:0/785028007 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce28005080 con 0x7fce40135b50 2024-09-17T15:54:15.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.775+0000 7fce4580f640 10 monclient: get_auth_request con 0x7fce24040cb0 auth_method 0 2024-09-17T15:54:15.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.779+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mon.8 v1:172.21.15.64:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fce34017860 con 0x7fce40135b50 2024-09-17T15:54:15.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.779+0000 7fce327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:15.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.779+0000 7fce327fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:15.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.961+0000 7fce47a73640 1 -- 172.21.15.64:0/785028007 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fce28000b90 con 0x7fce24040cb0 2024-09-17T15:54:15.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.962+0000 7fce327fc640 1 -- 172.21.15.64:0/785028007 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34003 (unknown 2965378022 0 419911493) 0x7fce28000b90 con 0x7fce24040cb0 2024-09-17T15:54:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:54:15.965 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-17T15:54:15.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.965+0000 7fce47a73640 1 -- 172.21.15.64:0/785028007 >> v1:172.21.15.100:6812/4210963484 conn(0x7fce24040cb0 legacy=0x7fce24043170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.965+0000 7fce47a73640 10 monclient: shutdown 2024-09-17T15:54:15.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.965+0000 7fce47a73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:15.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.965+0000 7fce47a73640 1 -- 172.21.15.64:0/785028007 >> v1:172.21.15.64:6793/0 conn(0x7fce40135b50 legacy=0x7fce401373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:15.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.965+0000 7fce47a73640 1 -- 172.21.15.64:0/785028007 shutdown_connections 2024-09-17T15:54:15.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.965+0000 7fce47a73640 1 -- 172.21.15.64:0/785028007 >> 172.21.15.64:0/785028007 conn(0x7fce4006db30 msgr2=0x7fce40133db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:15.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.965+0000 7fce47a73640 1 -- 172.21.15.64:0/785028007 shutdown_connections 2024-09-17T15:54:15.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:15.965+0000 7fce47a73640 1 -- 172.21.15.64:0/785028007 wait complete. 2024-09-17T15:54:15.977 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":573,"stamp":"2024-09-17T15:54:15.939984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":183872,"kb_used_data":4900,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566047168,"statfs":{"total":579820584960,"available":579632300032,"internally_reserved":0,"allocated":5017600,"data_stored":2520766,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":9568,"internal_metadata":182966944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002766"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":635,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.592448+0000","last_change":"2024-09-17T15:46:03.953098+0000","last_active":"2024-09-17T15:53:06.592448+0000","last_peered":"2024-09-17T15:53:06.592448+0000","last_clean":"2024-09-17T15:53:06.592448+0000","last_became_active":"2024-09-17T15:46:02.372737+0000","last_became_peered":"2024-09-17T15:46:02.372737+0000","last_unstale":"2024-09-17T15:53:06.592448+0000","last_undegraded":"2024-09-17T15:53:06.592448+0000","last_fullsized":"2024-09-17T15:53:06.592448+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:04:26.679936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057658000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":627,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.594073+0000","last_change":"2024-09-17T15:46:03.954609+0000","last_active":"2024-09-17T15:53:06.594073+0000","last_peered":"2024-09-17T15:53:06.594073+0000","last_clean":"2024-09-17T15:53:06.594073+0000","last_became_active":"2024-09-17T15:46:01.943967+0000","last_became_peered":"2024-09-17T15:46:01.943967+0000","last_unstale":"2024-09-17T15:53:06.594073+0000","last_undegraded":"2024-09-17T15:53:06.594073+0000","last_fullsized":"2024-09-17T15:53:06.594073+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:44:26.433116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058926900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":635,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.592297+0000","last_change":"2024-09-17T15:46:03.953301+0000","last_active":"2024-09-17T15:53:06.592297+0000","last_peered":"2024-09-17T15:53:06.592297+0000","last_clean":"2024-09-17T15:53:06.592297+0000","last_became_active":"2024-09-17T15:46:01.937233+0000","last_became_peered":"2024-09-17T15:46:01.937233+0000","last_unstale":"2024-09-17T15:53:06.592297+0000","last_undegraded":"2024-09-17T15:53:06.592297+0000","last_fullsized":"2024-09-17T15:53:06.592297+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:36:15.542838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001175981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":627,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.592454+0000","last_change":"2024-09-17T15:46:03.954824+0000","last_active":"2024-09-17T15:53:06.592454+0000","last_peered":"2024-09-17T15:53:06.592454+0000","last_clean":"2024-09-17T15:53:06.592454+0000","last_became_active":"2024-09-17T15:46:01.941419+0000","last_became_peered":"2024-09-17T15:46:01.941419+0000","last_unstale":"2024-09-17T15:53:06.592454+0000","last_undegraded":"2024-09-17T15:53:06.592454+0000","last_fullsized":"2024-09-17T15:53:06.592454+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:53:39.894895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077909099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":633,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.593738+0000","last_change":"2024-09-17T15:46:03.952714+0000","last_active":"2024-09-17T15:53:06.593738+0000","last_peered":"2024-09-17T15:53:06.593738+0000","last_clean":"2024-09-17T15:53:06.593738+0000","last_became_active":"2024-09-17T15:46:01.941164+0000","last_became_peered":"2024-09-17T15:46:01.941164+0000","last_unstale":"2024-09-17T15:53:06.593738+0000","last_undegraded":"2024-09-17T15:53:06.593738+0000","last_fullsized":"2024-09-17T15:53:06.593738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:07:59.957722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051738399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":629,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.709842+0000","last_change":"2024-09-17T15:46:03.954913+0000","last_active":"2024-09-17T15:53:06.709842+0000","last_peered":"2024-09-17T15:53:06.709842+0000","last_clean":"2024-09-17T15:53:06.709842+0000","last_became_active":"2024-09-17T15:46:01.941188+0000","last_became_peered":"2024-09-17T15:46:01.941188+0000","last_unstale":"2024-09-17T15:53:06.709842+0000","last_undegraded":"2024-09-17T15:53:06.709842+0000","last_fullsized":"2024-09-17T15:53:06.709842+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:25:32.291552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087819899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":636,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.591629+0000","last_change":"2024-09-17T15:46:03.953722+0000","last_active":"2024-09-17T15:53:06.591629+0000","last_peered":"2024-09-17T15:53:06.591629+0000","last_clean":"2024-09-17T15:53:06.591629+0000","last_became_active":"2024-09-17T15:46:01.941223+0000","last_became_peered":"2024-09-17T15:46:01.941223+0000","last_unstale":"2024-09-17T15:53:06.591629+0000","last_undegraded":"2024-09-17T15:53:06.591629+0000","last_fullsized":"2024-09-17T15:53:06.591629+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:18:15.696629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012017600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":632,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.594399+0000","last_change":"2024-09-17T15:46:03.952788+0000","last_active":"2024-09-17T15:53:06.594399+0000","last_peered":"2024-09-17T15:53:06.594399+0000","last_clean":"2024-09-17T15:53:06.594399+0000","last_became_active":"2024-09-17T15:46:01.937420+0000","last_became_peered":"2024-09-17T15:46:01.937420+0000","last_unstale":"2024-09-17T15:53:06.594399+0000","last_undegraded":"2024-09-17T15:53:06.594399+0000","last_fullsized":"2024-09-17T15:53:06.594399+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:03:18.015826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065159300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":737,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.591745+0000","last_change":"2024-09-17T15:45:56.919389+0000","last_active":"2024-09-17T15:53:06.591745+0000","last_peered":"2024-09-17T15:53:06.591745+0000","last_clean":"2024-09-17T15:53:06.591745+0000","last_became_active":"2024-09-17T15:45:56.908629+0000","last_became_peered":"2024-09-17T15:45:56.908629+0000","last_unstale":"2024-09-17T15:53:06.591745+0000","last_undegraded":"2024-09-17T15:53:06.591745+0000","last_fullsized":"2024-09-17T15:53:06.591745+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_clean_scrub_stamp":"2024-09-17T15:45:53.875287+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:44:47.696204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":1057856,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":14,"seq":60129542249,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":819200,"data_stored":243812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542248,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":823296,"data_stored":243831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542250,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":860160,"data_stored":772740,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":819200,"data_stored":243812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640360,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44480,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327360,"statfs":{"total":96636764160,"available":96591216640,"internally_reserved":0,"allocated":823296,"data_stored":243831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640360,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":872448,"data_stored":772740,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:54:15.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:54:16.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.077+0000 7fe8d53f5640 10 monclient: get_monmap_and_config 2024-09-17T15:54:16.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.077+0000 7fe8d53f5640 10 monclient: build_initial_monmap 2024-09-17T15:54:16.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.077+0000 7fe8d53f5640 10 monclient: monmap: 2024-09-17T15:54:16.079 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:16.079 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.079 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:16.078339+0000 2024-09-17T15:54:16.079 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:16.078339+0000 2024-09-17T15:54:16.079 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:16.079 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:16.080 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8d53f5640 1 Processor -- start 2024-09-17T15:54:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8d53f5640 1 -- start start 2024-09-17T15:54:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8d53f5640 10 monclient: init 2024-09-17T15:54:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8d53f5640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8d53f5640 10 monclient: _add_conns ranks=[7,2,6,1,3,0,8,4,5] 2024-09-17T15:54:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8d53f5640 10 monclient(hunting): picked mon.noname-i con 0x7fe8d00a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8d53f5640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d0135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8d53f5640 10 monclient(hunting): picked mon.noname-d con 0x7fe8d0135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8d53f5640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8d009d6f0 con 0x7fe8d0135b50 2024-09-17T15:54:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8d53f5640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8d009d9d0 con 0x7fe8d0135f70 2024-09-17T15:54:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8d53f5640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8d009dcb0 con 0x7fe8d00a5520 2024-09-17T15:54:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8d53f5640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8d53f5640 10 monclient(hunting): authenticate will time out at 1253.753676s 2024-09-17T15:54:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8cf7fe640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fe8d0135f70 0x7fe8d01415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:54818/0 (socket says 172.21.15.64:54818) 2024-09-17T15:54:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.078+0000 7fe8cf7fe640 1 -- 172.21.15.64:0/3120198417 learned_addr learned my addr 172.21.15.64:0/3120198417 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 1 -- 172.21.15.64:0/3120198417 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3000918765 0 0) 0x7fe8d009d6f0 con 0x7fe8d0135b50 2024-09-17T15:54:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient(hunting): my global_id is 125291 2024-09-17T15:54:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 1 -- 172.21.15.64:0/3120198417 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8b8003520 con 0x7fe8d0135b50 2024-09-17T15:54:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 1 -- 172.21.15.64:0/3120198417 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3271447245 0 0) 0x7fe8d009d9d0 con 0x7fe8d0135f70 2024-09-17T15:54:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient(hunting): my global_id is 174219 2024-09-17T15:54:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 1 -- 172.21.15.64:0/3120198417 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8d009d6f0 con 0x7fe8d0135f70 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 1 -- 172.21.15.64:0/3120198417 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 396292590 0 0) 0x7fe8b8003520 con 0x7fe8d0135b50 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 1 -- 172.21.15.64:0/3120198417 >> v1:172.21.15.100:6792/0 conn(0x7fe8d00a5520 legacy=0x7fe8d00a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 1 -- 172.21.15.64:0/3120198417 >> v1:172.21.15.64:6792/0 conn(0x7fe8d0135f70 legacy=0x7fe8d01415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient: _finish_hunting 0 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 1 monclient: found mon.noname-b 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 1 -- 172.21.15.64:0/3120198417 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d009ccd0 con 0x7fe8d0135b50 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient: _finish_auth 0 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient: _check_auth_tickets 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 1 -- 172.21.15.64:0/3120198417 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe8c4001830 con 0x7fe8d0135b50 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8d53f5640 5 monclient: authenticate success, global_id 125291 2024-09-17T15:54:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8d53f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient: dump: 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.090 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.091 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.091 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.091 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.091 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.091 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8d53f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:16.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 1 -- 172.21.15.64:0/3120198417 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe8c40019d0 con 0x7fe8d0135b50 2024-09-17T15:54:16.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:16.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 1 -- 172.21.15.64:0/3120198417 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe8c4003950 con 0x7fe8d0135b50 2024-09-17T15:54:16.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:16.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:16.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8cdffb640 10 monclient: dump: 2024-09-17T15:54:16.092 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.092 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.092 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.092 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.092 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.092 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.092 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.092 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.092 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.092 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.092 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8d53f5640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8d53f5640 4 set_mon_vals no callback set 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8d53f5640 10 monclient: shutdown 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8d53f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8d53f5640 1 -- 172.21.15.64:0/3120198417 >> v1:172.21.15.64:6790/0 conn(0x7fe8d0135b50 legacy=0x7fe8d013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8d53f5640 1 -- 172.21.15.64:0/3120198417 shutdown_connections 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8d53f5640 1 -- 172.21.15.64:0/3120198417 >> 172.21.15.64:0/3120198417 conn(0x7fe8d006db30 msgr2=0x7fe8d00ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.079+0000 7fe8d53f5640 1 -- 172.21.15.64:0/3120198417 shutdown_connections 2024-09-17T15:54:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.080+0000 7fe8d53f5640 1 -- 172.21.15.64:0/3120198417 wait complete. 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.080+0000 7fe8d53f5640 10 monclient: build_initial_monmap 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.080+0000 7fe8d53f5640 10 monclient: monmap: 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.080+0000 7fe8d53f5640 1 Processor -- start 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8d53f5640 1 -- start start 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8d53f5640 10 monclient: init 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8d53f5640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8d53f5640 10 monclient: _add_conns ranks=[7,4,0,3,8,2,1,6,5] 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8d53f5640 10 monclient(hunting): picked mon.noname-i con 0x7fe8d00a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8d53f5640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d0135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8d53f5640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d0135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8d53f5640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8d00a08e0 con 0x7fe8d0135f70 2024-09-17T15:54:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8d53f5640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8d01352f0 con 0x7fe8d0135b50 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8d53f5640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8d0134f30 con 0x7fe8d00a5520 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8d53f5640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8d53f5640 10 monclient(hunting): authenticate will time out at 1253.756513s 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8ce7fc640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fe8d0135b50 0x7fe8d01320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:59838/0 (socket says 172.21.15.64:59838) 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8ce7fc640 1 -- 172.21.15.64:0/4129399908 learned_addr learned my addr 172.21.15.64:0/4129399908 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 663088627 0 0) 0x7fe8d01352f0 con 0x7fe8d0135b50 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 10 monclient(hunting): my global_id is 154714 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8bc003520 con 0x7fe8d0135b50 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 370259045 0 0) 0x7fe8d00a08e0 con 0x7fe8d0135f70 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 10 monclient(hunting): my global_id is 125154 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8d01352f0 con 0x7fe8d0135f70 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1110341633 0 0) 0x7fe8d0134f30 con 0x7fe8d00a5520 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 10 monclient(hunting): my global_id is 174193 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.081+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8d00a08e0 con 0x7fe8d00a5520 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3348457371 0 0) 0x7fe8bc003520 con 0x7fe8d0135b50 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8d0134f30 con 0x7fe8d0135b50 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe8c4001430 con 0x7fe8d0135b50 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:16.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient(hunting): dump: 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.098 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2428774482 0 0) 0x7fe8d01352f0 con 0x7fe8d0135f70 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8bc003520 con 0x7fe8d0135f70 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe8c8001cf0 con 0x7fe8d0135f70 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient(hunting): dump: 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3732855286 0 0) 0x7fe8d00a08e0 con 0x7fe8d00a5520 2024-09-17T15:54:16.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8d01352f0 con 0x7fe8d00a5520 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe8c0001a10 con 0x7fe8d00a5520 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient(hunting): dump: 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1835341332 0 0) 0x7fe8d0134f30 con 0x7fe8d0135b50 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 >> v1:172.21.15.100:6792/0 conn(0x7fe8d00a5520 legacy=0x7fe8d01319b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 >> v1:172.21.15.64:6789/0 conn(0x7fe8d0135f70 legacy=0x7fe8d01d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 monclient: found mon.c 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d009fa70 con 0x7fe8d0135b50 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient: _finish_auth 0 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8d53f5640 5 monclient: authenticate success, global_id 154714 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8d53f5640 10 monclient: _renew_subs 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8d53f5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8d53f5640 1 -- 172.21.15.64:0/4129399908 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe8d01327d0 con 0x7fe8d0135b50 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8d53f5640 10 monclient: _renew_subs 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8d53f5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8d53f5640 1 -- 172.21.15.64:0/4129399908 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fe8d009f380 con 0x7fe8d0135b50 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe8c4003c10 con 0x7fe8d0135b50 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fe8c4004200 con 0x7fe8d0135b50 2024-09-17T15:54:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8cd7fa640 4 set_mon_vals no callback set 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.082+0000 7fe8b77fe640 10 monclient: dump: 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.105 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.083+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fe8c40043a0 con 0x7fe8d0135b50 2024-09-17T15:54:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.083+0000 7fe8d53f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.083+0000 7fe8d53f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.083+0000 7fe8d53f5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.083+0000 7fe8d53f5640 1 -- 172.21.15.64:0/4129399908 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8b0005080 con 0x7fe8d0135b50 2024-09-17T15:54:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.083+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7fe8c404eb80 con 0x7fe8d0135b50 2024-09-17T15:54:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.084+0000 7fe8ceffd640 10 monclient: get_auth_request con 0x7fe8bc040d20 auth_method 0 2024-09-17T15:54:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.088+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fe8c40178f0 con 0x7fe8d0135b50 2024-09-17T15:54:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.088+0000 7fe8b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.088+0000 7fe8b77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:16.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.269+0000 7fe8d53f5640 1 -- 172.21.15.64:0/4129399908 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe8b0000b90 con 0x7fe8bc040d20 2024-09-17T15:54:16.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.270+0000 7fe8b77fe640 1 -- 172.21.15.64:0/4129399908 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34003 (unknown 2965378022 0 419911493) 0x7fe8b0000b90 con 0x7fe8bc040d20 2024-09-17T15:54:16.272 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:54:16.273 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-17T15:54:16.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.272+0000 7fe8d53f5640 1 -- 172.21.15.64:0/4129399908 >> v1:172.21.15.100:6812/4210963484 conn(0x7fe8bc040d20 legacy=0x7fe8bc0431e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.272+0000 7fe8d53f5640 10 monclient: shutdown 2024-09-17T15:54:16.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.272+0000 7fe8d53f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:16.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.272+0000 7fe8d53f5640 1 -- 172.21.15.64:0/4129399908 >> v1:172.21.15.64:6791/0 conn(0x7fe8d0135b50 legacy=0x7fe8d01320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.272+0000 7fe8d53f5640 1 -- 172.21.15.64:0/4129399908 shutdown_connections 2024-09-17T15:54:16.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.272+0000 7fe8d53f5640 1 -- 172.21.15.64:0/4129399908 >> 172.21.15.64:0/4129399908 conn(0x7fe8d006db30 msgr2=0x7fe8d01334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:16.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.272+0000 7fe8d53f5640 1 -- 172.21.15.64:0/4129399908 shutdown_connections 2024-09-17T15:54:16.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.272+0000 7fe8d53f5640 1 -- 172.21.15.64:0/4129399908 wait complete. 2024-09-17T15:54:16.285 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":573,"stamp":"2024-09-17T15:54:15.939984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":183872,"kb_used_data":4900,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566047168,"statfs":{"total":579820584960,"available":579632300032,"internally_reserved":0,"allocated":5017600,"data_stored":2520766,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":9568,"internal_metadata":182966944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002766"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":635,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.592448+0000","last_change":"2024-09-17T15:46:03.953098+0000","last_active":"2024-09-17T15:53:06.592448+0000","last_peered":"2024-09-17T15:53:06.592448+0000","last_clean":"2024-09-17T15:53:06.592448+0000","last_became_active":"2024-09-17T15:46:02.372737+0000","last_became_peered":"2024-09-17T15:46:02.372737+0000","last_unstale":"2024-09-17T15:53:06.592448+0000","last_undegraded":"2024-09-17T15:53:06.592448+0000","last_fullsized":"2024-09-17T15:53:06.592448+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:04:26.679936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057658000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":627,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.594073+0000","last_change":"2024-09-17T15:46:03.954609+0000","last_active":"2024-09-17T15:53:06.594073+0000","last_peered":"2024-09-17T15:53:06.594073+0000","last_clean":"2024-09-17T15:53:06.594073+0000","last_became_active":"2024-09-17T15:46:01.943967+0000","last_became_peered":"2024-09-17T15:46:01.943967+0000","last_unstale":"2024-09-17T15:53:06.594073+0000","last_undegraded":"2024-09-17T15:53:06.594073+0000","last_fullsized":"2024-09-17T15:53:06.594073+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:44:26.433116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058926900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":635,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.592297+0000","last_change":"2024-09-17T15:46:03.953301+0000","last_active":"2024-09-17T15:53:06.592297+0000","last_peered":"2024-09-17T15:53:06.592297+0000","last_clean":"2024-09-17T15:53:06.592297+0000","last_became_active":"2024-09-17T15:46:01.937233+0000","last_became_peered":"2024-09-17T15:46:01.937233+0000","last_unstale":"2024-09-17T15:53:06.592297+0000","last_undegraded":"2024-09-17T15:53:06.592297+0000","last_fullsized":"2024-09-17T15:53:06.592297+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:36:15.542838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001175981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":627,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.592454+0000","last_change":"2024-09-17T15:46:03.954824+0000","last_active":"2024-09-17T15:53:06.592454+0000","last_peered":"2024-09-17T15:53:06.592454+0000","last_clean":"2024-09-17T15:53:06.592454+0000","last_became_active":"2024-09-17T15:46:01.941419+0000","last_became_peered":"2024-09-17T15:46:01.941419+0000","last_unstale":"2024-09-17T15:53:06.592454+0000","last_undegraded":"2024-09-17T15:53:06.592454+0000","last_fullsized":"2024-09-17T15:53:06.592454+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:53:39.894895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077909099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":633,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.593738+0000","last_change":"2024-09-17T15:46:03.952714+0000","last_active":"2024-09-17T15:53:06.593738+0000","last_peered":"2024-09-17T15:53:06.593738+0000","last_clean":"2024-09-17T15:53:06.593738+0000","last_became_active":"2024-09-17T15:46:01.941164+0000","last_became_peered":"2024-09-17T15:46:01.941164+0000","last_unstale":"2024-09-17T15:53:06.593738+0000","last_undegraded":"2024-09-17T15:53:06.593738+0000","last_fullsized":"2024-09-17T15:53:06.593738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:07:59.957722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051738399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":629,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.709842+0000","last_change":"2024-09-17T15:46:03.954913+0000","last_active":"2024-09-17T15:53:06.709842+0000","last_peered":"2024-09-17T15:53:06.709842+0000","last_clean":"2024-09-17T15:53:06.709842+0000","last_became_active":"2024-09-17T15:46:01.941188+0000","last_became_peered":"2024-09-17T15:46:01.941188+0000","last_unstale":"2024-09-17T15:53:06.709842+0000","last_undegraded":"2024-09-17T15:53:06.709842+0000","last_fullsized":"2024-09-17T15:53:06.709842+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:25:32.291552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087819899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":636,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.591629+0000","last_change":"2024-09-17T15:46:03.953722+0000","last_active":"2024-09-17T15:53:06.591629+0000","last_peered":"2024-09-17T15:53:06.591629+0000","last_clean":"2024-09-17T15:53:06.591629+0000","last_became_active":"2024-09-17T15:46:01.941223+0000","last_became_peered":"2024-09-17T15:46:01.941223+0000","last_unstale":"2024-09-17T15:53:06.591629+0000","last_undegraded":"2024-09-17T15:53:06.591629+0000","last_fullsized":"2024-09-17T15:53:06.591629+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:18:15.696629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012017600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":632,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.594399+0000","last_change":"2024-09-17T15:46:03.952788+0000","last_active":"2024-09-17T15:53:06.594399+0000","last_peered":"2024-09-17T15:53:06.594399+0000","last_clean":"2024-09-17T15:53:06.594399+0000","last_became_active":"2024-09-17T15:46:01.937420+0000","last_became_peered":"2024-09-17T15:46:01.937420+0000","last_unstale":"2024-09-17T15:53:06.594399+0000","last_undegraded":"2024-09-17T15:53:06.594399+0000","last_fullsized":"2024-09-17T15:53:06.594399+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:03:18.015826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065159300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":737,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.591745+0000","last_change":"2024-09-17T15:45:56.919389+0000","last_active":"2024-09-17T15:53:06.591745+0000","last_peered":"2024-09-17T15:53:06.591745+0000","last_clean":"2024-09-17T15:53:06.591745+0000","last_became_active":"2024-09-17T15:45:56.908629+0000","last_became_peered":"2024-09-17T15:45:56.908629+0000","last_unstale":"2024-09-17T15:53:06.591745+0000","last_undegraded":"2024-09-17T15:53:06.591745+0000","last_fullsized":"2024-09-17T15:53:06.591745+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_clean_scrub_stamp":"2024-09-17T15:45:53.875287+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:44:47.696204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":1057856,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":14,"seq":60129542249,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":819200,"data_stored":243812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542248,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":823296,"data_stored":243831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542250,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":860160,"data_stored":772740,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":819200,"data_stored":243812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640360,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44480,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327360,"statfs":{"total":96636764160,"available":96591216640,"internally_reserved":0,"allocated":823296,"data_stored":243831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640360,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":872448,"data_stored":772740,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:54:16.286 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T15:54:16.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:54:16.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.385+0000 7f9acce7c640 10 monclient: get_monmap_and_config 2024-09-17T15:54:16.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.385+0000 7f9acce7c640 10 monclient: build_initial_monmap 2024-09-17T15:54:16.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.385+0000 7f9acce7c640 10 monclient: monmap: 2024-09-17T15:54:16.387 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:16.387 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.387 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:16.386042+0000 2024-09-17T15:54:16.387 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:16.386042+0000 2024-09-17T15:54:16.387 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:16.387 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.387 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:16.387 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:16.387 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:16.388 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:16.388 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:16.388 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:16.388 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:16.388 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:16.388 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:16.388 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.385+0000 7f9acce7c640 1 Processor -- start 2024-09-17T15:54:16.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.385+0000 7f9acce7c640 1 -- start start 2024-09-17T15:54:16.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.385+0000 7f9acce7c640 10 monclient: init 2024-09-17T15:54:16.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9acce7c640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:16.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9acce7c640 10 monclient: _add_conns ranks=[3,1,6,4,5,0,2,8,7] 2024-09-17T15:54:16.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9acce7c640 10 monclient(hunting): picked mon.noname-g con 0x7f9ac80a54b0 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:16.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9acce7c640 10 monclient(hunting): picked mon.noname-f con 0x7f9ac8135c80 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:16.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9acce7c640 10 monclient(hunting): picked mon.noname-d con 0x7f9ac81360a0 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:16.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9acce7c640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac80aacb0 con 0x7f9ac8135c80 2024-09-17T15:54:16.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9acce7c640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac809d780 con 0x7f9ac80a54b0 2024-09-17T15:54:16.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9acce7c640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac809da60 con 0x7f9ac81360a0 2024-09-17T15:54:16.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9acce7c640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:16.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9acce7c640 10 monclient(hunting): authenticate will time out at 1254.061571s 2024-09-17T15:54:16.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9ac6d9d640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f9ac81360a0 0x7f9ac8141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:54822/0 (socket says 172.21.15.64:54822) 2024-09-17T15:54:16.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9ac6d9d640 1 -- 172.21.15.64:0/2350146666 learned_addr learned my addr 172.21.15.64:0/2350146666 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:16.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9ac559a640 1 -- 172.21.15.64:0/2350146666 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3178064328 0 0) 0x7f9ac809da60 con 0x7f9ac81360a0 2024-09-17T15:54:16.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9ac559a640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9ac559a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9ac559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9ac559a640 10 monclient(hunting): my global_id is 174228 2024-09-17T15:54:16.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.386+0000 7f9ac559a640 1 -- 172.21.15.64:0/2350146666 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ab4003520 con 0x7f9ac81360a0 2024-09-17T15:54:16.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 1 -- 172.21.15.64:0/2350146666 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3653232053 0 0) 0x7f9ac80aacb0 con 0x7f9ac8135c80 2024-09-17T15:54:16.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient(hunting): my global_id is 6958 2024-09-17T15:54:16.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 1 -- 172.21.15.64:0/2350146666 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ac809da60 con 0x7f9ac8135c80 2024-09-17T15:54:16.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 1 -- 172.21.15.64:0/2350146666 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 648097328 0 0) 0x7f9ac809d780 con 0x7f9ac80a54b0 2024-09-17T15:54:16.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient(hunting): my global_id is 115950 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 1 -- 172.21.15.64:0/2350146666 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ac80aacb0 con 0x7f9ac80a54b0 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 1 -- 172.21.15.64:0/2350146666 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2071843157 0 0) 0x7f9ab4003520 con 0x7f9ac81360a0 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 1 -- 172.21.15.64:0/2350146666 >> v1:172.21.15.100:6790/0 conn(0x7f9ac80a54b0 legacy=0x7f9ac80a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 1 -- 172.21.15.64:0/2350146666 >> v1:172.21.15.100:6789/0 conn(0x7f9ac8135c80 legacy=0x7f9ac813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient: _finish_hunting 0 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 1 monclient: found mon.noname-d 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 1 -- 172.21.15.64:0/2350146666 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac809dd40 con 0x7f9ac81360a0 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient: _finish_auth 0 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient: _check_auth_tickets 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:16.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 1 -- 172.21.15.64:0/2350146666 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9ac0001bb0 con 0x7f9ac81360a0 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient: dump: 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9acce7c640 5 monclient: authenticate success, global_id 174228 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9acce7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 1 -- 172.21.15.64:0/2350146666 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9ac0001dd0 con 0x7f9ac81360a0 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9acce7c640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 1 -- 172.21.15.64:0/2350146666 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9ac0004f60 con 0x7f9ac81360a0 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9acce7c640 4 set_mon_vals no callback set 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:16.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9ac559a640 10 monclient: dump: 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.400 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.401 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.401 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.401 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9acce7c640 10 monclient: shutdown 2024-09-17T15:54:16.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9acce7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:16.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9acce7c640 1 -- 172.21.15.64:0/2350146666 >> v1:172.21.15.64:6792/0 conn(0x7f9ac81360a0 legacy=0x7f9ac8141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9acce7c640 1 -- 172.21.15.64:0/2350146666 shutdown_connections 2024-09-17T15:54:16.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9acce7c640 1 -- 172.21.15.64:0/2350146666 >> 172.21.15.64:0/2350146666 conn(0x7f9ac806daa0 msgr2=0x7f9ac80ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:16.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9acce7c640 1 -- 172.21.15.64:0/2350146666 shutdown_connections 2024-09-17T15:54:16.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.387+0000 7f9acce7c640 1 -- 172.21.15.64:0/2350146666 wait complete. 2024-09-17T15:54:16.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.388+0000 7f9acce7c640 10 monclient: build_initial_monmap 2024-09-17T15:54:16.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.388+0000 7f9acce7c640 10 monclient: monmap: 2024-09-17T15:54:16.401 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:16.402 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:16.402 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:16.402 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:16.402 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:16.402 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:16.402 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:16.402 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:16.402 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:16.402 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:16.402 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:16.402 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:16.402 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:16.402 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:16.403 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:16.403 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.388+0000 7f9acce7c640 1 Processor -- start 2024-09-17T15:54:16.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9acce7c640 1 -- start start 2024-09-17T15:54:16.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9acce7c640 10 monclient: init 2024-09-17T15:54:16.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9acce7c640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:16.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9acce7c640 10 monclient: _add_conns ranks=[8,0,3,6,2,1,5,4,7] 2024-09-17T15:54:16.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9acce7c640 10 monclient(hunting): picked mon.noname-e con 0x7f9ac80a54b0 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:16.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9acce7c640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac8135c80 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:16.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9acce7c640 10 monclient(hunting): picked mon.noname-g con 0x7f9ac81360a0 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:16.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9acce7c640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac80704c0 con 0x7f9ac8135c80 2024-09-17T15:54:16.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9acce7c640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac8123120 con 0x7f9ac81360a0 2024-09-17T15:54:16.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9acce7c640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac8123400 con 0x7f9ac80a54b0 2024-09-17T15:54:16.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9acce7c640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:16.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9acce7c640 10 monclient(hunting): authenticate will time out at 1254.064472s 2024-09-17T15:54:16.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9ac659c640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f9ac80a54b0 0x7f9ac81ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51084/0 (socket says 172.21.15.64:51084) 2024-09-17T15:54:16.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9ac659c640 1 -- 172.21.15.64:0/1077386491 learned_addr learned my addr 172.21.15.64:0/1077386491 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:16.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 244858762 0 0) 0x7f9ac80704c0 con 0x7f9ac8135c80 2024-09-17T15:54:16.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 10 monclient(hunting): my global_id is 125163 2024-09-17T15:54:16.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9aa8003520 con 0x7f9ac8135c80 2024-09-17T15:54:16.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1189075198 0 0) 0x7f9ac8123400 con 0x7f9ac80a54b0 2024-09-17T15:54:16.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 10 monclient(hunting): my global_id is 174203 2024-09-17T15:54:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ac80704c0 con 0x7f9ac80a54b0 2024-09-17T15:54:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3898185485 0 0) 0x7f9ac8123120 con 0x7f9ac81360a0 2024-09-17T15:54:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 10 monclient(hunting): my global_id is 115959 2024-09-17T15:54:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.389+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ac8123400 con 0x7f9ac81360a0 2024-09-17T15:54:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3388636141 0 0) 0x7f9aa8003520 con 0x7f9ac8135c80 2024-09-17T15:54:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ac8123120 con 0x7f9ac8135c80 2024-09-17T15:54:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9ab0001a50 con 0x7f9ac8135c80 2024-09-17T15:54:16.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:16.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:54:16.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient(hunting): dump: 2024-09-17T15:54:16.406 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.406 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.406 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.406 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.406 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.406 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.406 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.406 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.406 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.406 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.406 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.407 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.407 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.407 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.407 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.407 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:16.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2461096397 0 0) 0x7f9ac80704c0 con 0x7f9ac80a54b0 2024-09-17T15:54:16.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9aa8003520 con 0x7f9ac80a54b0 2024-09-17T15:54:16.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9ab8001b90 con 0x7f9ac80a54b0 2024-09-17T15:54:16.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:16.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:16.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient(hunting): dump: 2024-09-17T15:54:16.407 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.408 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.408 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.408 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.408 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.408 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.408 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.408 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.408 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.408 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.408 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.408 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.408 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.408 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.409 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.409 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2520329209 0 0) 0x7f9ac8123400 con 0x7f9ac81360a0 2024-09-17T15:54:16.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ac80704c0 con 0x7f9ac81360a0 2024-09-17T15:54:16.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1903564762 0 0) 0x7f9ac8123120 con 0x7f9ac8135c80 2024-09-17T15:54:16.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 >> v1:172.21.15.64:6793/0 conn(0x7f9ac80a54b0 legacy=0x7f9ac81ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 >> v1:172.21.15.100:6790/0 conn(0x7f9ac81360a0 legacy=0x7f9ac81d6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient: _finish_hunting 0 2024-09-17T15:54:16.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 monclient: found mon.a 2024-09-17T15:54:16.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:16.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:16.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac8122000 con 0x7f9ac8135c80 2024-09-17T15:54:16.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient: _finish_auth 0 2024-09-17T15:54:16.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:54:16.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:16.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9acce7c640 5 monclient: authenticate success, global_id 125163 2024-09-17T15:54:16.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9acce7c640 10 monclient: _renew_subs 2024-09-17T15:54:16.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9acce7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:16.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9acce7c640 1 -- 172.21.15.64:0/1077386491 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac8132860 con 0x7f9ac8135c80 2024-09-17T15:54:16.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9acce7c640 10 monclient: _renew_subs 2024-09-17T15:54:16.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9acce7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:16.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9acce7c640 1 -- 172.21.15.64:0/1077386491 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9ac8132170 con 0x7f9ac8135c80 2024-09-17T15:54:16.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9ab00038c0 con 0x7f9ac8135c80 2024-09-17T15:54:16.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:16.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f9ab0003db0 con 0x7f9ac8135c80 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9ac4d99640 4 set_mon_vals no callback set 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.390+0000 7f9abeffd640 10 monclient: dump: 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.411 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.412 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.412 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.412 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.412 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.412 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.412 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.391+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f9ab0010690 con 0x7f9ac8135c80 2024-09-17T15:54:16.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.391+0000 7f9acce7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:16.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.391+0000 7f9acce7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:16.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.391+0000 7f9acce7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:16.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.391+0000 7f9acce7c640 1 -- 172.21.15.64:0/1077386491 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aac005080 con 0x7f9ac8135c80 2024-09-17T15:54:16.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.391+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f9ab004d830 con 0x7f9ac8135c80 2024-09-17T15:54:16.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.391+0000 7f9ac659c640 10 monclient: get_auth_request con 0x7f9aa80409d0 auth_method 0 2024-09-17T15:54:16.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.396+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f9ab0016620 con 0x7f9ac8135c80 2024-09-17T15:54:16.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.396+0000 7f9abeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:16.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.396+0000 7f9abeffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:16.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.586+0000 7f9acce7c640 1 -- 172.21.15.64:0/1077386491 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9aac000b90 con 0x7f9aa80409d0 2024-09-17T15:54:16.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.587+0000 7f9abeffd640 1 -- 172.21.15.64:0/1077386491 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34003 (unknown 2965378022 0 419911493) 0x7f9aac000b90 con 0x7f9aa80409d0 2024-09-17T15:54:16.589 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:54:16.590 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-17T15:54:16.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.589+0000 7f9acce7c640 1 -- 172.21.15.64:0/1077386491 >> v1:172.21.15.100:6812/4210963484 conn(0x7f9aa80409d0 legacy=0x7f9aa8042e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.589+0000 7f9acce7c640 10 monclient: shutdown 2024-09-17T15:54:16.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.589+0000 7f9acce7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:16.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.589+0000 7f9acce7c640 1 -- 172.21.15.64:0/1077386491 >> v1:172.21.15.64:6789/0 conn(0x7f9ac8135c80 legacy=0x7f9ac81d3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.589+0000 7f9acce7c640 1 -- 172.21.15.64:0/1077386491 shutdown_connections 2024-09-17T15:54:16.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.589+0000 7f9acce7c640 1 -- 172.21.15.64:0/1077386491 >> 172.21.15.64:0/1077386491 conn(0x7f9ac806daa0 msgr2=0x7f9ac812c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:16.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.589+0000 7f9acce7c640 1 -- 172.21.15.64:0/1077386491 shutdown_connections 2024-09-17T15:54:16.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.589+0000 7f9acce7c640 1 -- 172.21.15.64:0/1077386491 wait complete. 2024-09-17T15:54:16.601 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":573,"stamp":"2024-09-17T15:54:15.939984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":183872,"kb_used_data":4900,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566047168,"statfs":{"total":579820584960,"available":579632300032,"internally_reserved":0,"allocated":5017600,"data_stored":2520766,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":9568,"internal_metadata":182966944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002766"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":635,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.592448+0000","last_change":"2024-09-17T15:46:03.953098+0000","last_active":"2024-09-17T15:53:06.592448+0000","last_peered":"2024-09-17T15:53:06.592448+0000","last_clean":"2024-09-17T15:53:06.592448+0000","last_became_active":"2024-09-17T15:46:02.372737+0000","last_became_peered":"2024-09-17T15:46:02.372737+0000","last_unstale":"2024-09-17T15:53:06.592448+0000","last_undegraded":"2024-09-17T15:53:06.592448+0000","last_fullsized":"2024-09-17T15:53:06.592448+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:04:26.679936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057658000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":627,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.594073+0000","last_change":"2024-09-17T15:46:03.954609+0000","last_active":"2024-09-17T15:53:06.594073+0000","last_peered":"2024-09-17T15:53:06.594073+0000","last_clean":"2024-09-17T15:53:06.594073+0000","last_became_active":"2024-09-17T15:46:01.943967+0000","last_became_peered":"2024-09-17T15:46:01.943967+0000","last_unstale":"2024-09-17T15:53:06.594073+0000","last_undegraded":"2024-09-17T15:53:06.594073+0000","last_fullsized":"2024-09-17T15:53:06.594073+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:44:26.433116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058926900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":635,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.592297+0000","last_change":"2024-09-17T15:46:03.953301+0000","last_active":"2024-09-17T15:53:06.592297+0000","last_peered":"2024-09-17T15:53:06.592297+0000","last_clean":"2024-09-17T15:53:06.592297+0000","last_became_active":"2024-09-17T15:46:01.937233+0000","last_became_peered":"2024-09-17T15:46:01.937233+0000","last_unstale":"2024-09-17T15:53:06.592297+0000","last_undegraded":"2024-09-17T15:53:06.592297+0000","last_fullsized":"2024-09-17T15:53:06.592297+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:36:15.542838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001175981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":627,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.592454+0000","last_change":"2024-09-17T15:46:03.954824+0000","last_active":"2024-09-17T15:53:06.592454+0000","last_peered":"2024-09-17T15:53:06.592454+0000","last_clean":"2024-09-17T15:53:06.592454+0000","last_became_active":"2024-09-17T15:46:01.941419+0000","last_became_peered":"2024-09-17T15:46:01.941419+0000","last_unstale":"2024-09-17T15:53:06.592454+0000","last_undegraded":"2024-09-17T15:53:06.592454+0000","last_fullsized":"2024-09-17T15:53:06.592454+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:53:39.894895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077909099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":633,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.593738+0000","last_change":"2024-09-17T15:46:03.952714+0000","last_active":"2024-09-17T15:53:06.593738+0000","last_peered":"2024-09-17T15:53:06.593738+0000","last_clean":"2024-09-17T15:53:06.593738+0000","last_became_active":"2024-09-17T15:46:01.941164+0000","last_became_peered":"2024-09-17T15:46:01.941164+0000","last_unstale":"2024-09-17T15:53:06.593738+0000","last_undegraded":"2024-09-17T15:53:06.593738+0000","last_fullsized":"2024-09-17T15:53:06.593738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:07:59.957722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051738399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":629,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.709842+0000","last_change":"2024-09-17T15:46:03.954913+0000","last_active":"2024-09-17T15:53:06.709842+0000","last_peered":"2024-09-17T15:53:06.709842+0000","last_clean":"2024-09-17T15:53:06.709842+0000","last_became_active":"2024-09-17T15:46:01.941188+0000","last_became_peered":"2024-09-17T15:46:01.941188+0000","last_unstale":"2024-09-17T15:53:06.709842+0000","last_undegraded":"2024-09-17T15:53:06.709842+0000","last_fullsized":"2024-09-17T15:53:06.709842+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:25:32.291552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087819899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":636,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.591629+0000","last_change":"2024-09-17T15:46:03.953722+0000","last_active":"2024-09-17T15:53:06.591629+0000","last_peered":"2024-09-17T15:53:06.591629+0000","last_clean":"2024-09-17T15:53:06.591629+0000","last_became_active":"2024-09-17T15:46:01.941223+0000","last_became_peered":"2024-09-17T15:46:01.941223+0000","last_unstale":"2024-09-17T15:53:06.591629+0000","last_undegraded":"2024-09-17T15:53:06.591629+0000","last_fullsized":"2024-09-17T15:53:06.591629+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:18:15.696629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012017600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":632,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.594399+0000","last_change":"2024-09-17T15:46:03.952788+0000","last_active":"2024-09-17T15:53:06.594399+0000","last_peered":"2024-09-17T15:53:06.594399+0000","last_clean":"2024-09-17T15:53:06.594399+0000","last_became_active":"2024-09-17T15:46:01.937420+0000","last_became_peered":"2024-09-17T15:46:01.937420+0000","last_unstale":"2024-09-17T15:53:06.594399+0000","last_undegraded":"2024-09-17T15:53:06.594399+0000","last_fullsized":"2024-09-17T15:53:06.594399+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:03:18.015826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065159300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":737,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.591745+0000","last_change":"2024-09-17T15:45:56.919389+0000","last_active":"2024-09-17T15:53:06.591745+0000","last_peered":"2024-09-17T15:53:06.591745+0000","last_clean":"2024-09-17T15:53:06.591745+0000","last_became_active":"2024-09-17T15:45:56.908629+0000","last_became_peered":"2024-09-17T15:45:56.908629+0000","last_unstale":"2024-09-17T15:53:06.591745+0000","last_undegraded":"2024-09-17T15:53:06.591745+0000","last_fullsized":"2024-09-17T15:53:06.591745+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_clean_scrub_stamp":"2024-09-17T15:45:53.875287+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:44:47.696204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":1057856,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":14,"seq":60129542249,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":819200,"data_stored":243812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542248,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":823296,"data_stored":243831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542250,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":860160,"data_stored":772740,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":819200,"data_stored":243812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640360,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44480,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327360,"statfs":{"total":96636764160,"available":96591216640,"internally_reserved":0,"allocated":823296,"data_stored":243831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640360,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":872448,"data_stored":772740,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:54:16.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T15:54:16.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.701+0000 7f55afe47640 10 monclient: get_monmap_and_config 2024-09-17T15:54:16.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.701+0000 7f55afe47640 10 monclient: build_initial_monmap 2024-09-17T15:54:16.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.701+0000 7f55afe47640 10 monclient: monmap: 2024-09-17T15:54:16.703 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:16.703 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:16.702545+0000 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:16.702545+0000 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:16.704 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.702+0000 7f55afe47640 1 Processor -- start 2024-09-17T15:54:16.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.702+0000 7f55afe47640 1 -- start start 2024-09-17T15:54:16.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.702+0000 7f55afe47640 10 monclient: init 2024-09-17T15:54:16.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.702+0000 7f55afe47640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:16.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 10 monclient: _add_conns ranks=[8,6,7,4,3,2,0,5,1] 2024-09-17T15:54:16.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 10 monclient(hunting): picked mon.noname-e con 0x7f55a80a54b0 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:16.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 10 monclient(hunting): picked mon.noname-d con 0x7f55a8135c80 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:16.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 10 monclient(hunting): picked mon.noname-i con 0x7f55a81360a0 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:16.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55a80aacb0 con 0x7f55a8135c80 2024-09-17T15:54:16.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55a809d780 con 0x7f55a81360a0 2024-09-17T15:54:16.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55a809da60 con 0x7f55a80a54b0 2024-09-17T15:54:16.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:16.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 10 monclient(hunting): authenticate will time out at 1254.378170s 2024-09-17T15:54:16.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55adbe3640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f55a80a54b0 0x7f55a80a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51090/0 (socket says 172.21.15.64:51090) 2024-09-17T15:54:16.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55adbe3640 1 -- 172.21.15.64:0/4023777184 learned_addr learned my addr 172.21.15.64:0/4023777184 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:16.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 1 -- 172.21.15.64:0/4023777184 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2052165298 0 0) 0x7f55a80aacb0 con 0x7f55a8135c80 2024-09-17T15:54:16.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient(hunting): my global_id is 174237 2024-09-17T15:54:16.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 1 -- 172.21.15.64:0/4023777184 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5594003520 con 0x7f55a8135c80 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 1 -- 172.21.15.64:0/4023777184 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4236018488 0 0) 0x7f55a809da60 con 0x7f55a80a54b0 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient(hunting): my global_id is 174212 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 1 -- 172.21.15.64:0/4023777184 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55a80aacb0 con 0x7f55a80a54b0 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 1 -- 172.21.15.64:0/4023777184 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1169295449 0 0) 0x7f5594003520 con 0x7f55a8135c80 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 1 -- 172.21.15.64:0/4023777184 >> v1:172.21.15.64:6793/0 conn(0x7f55a80a54b0 legacy=0x7f55a80a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 1 -- 172.21.15.64:0/4023777184 >> v1:172.21.15.100:6792/0 conn(0x7f55a81360a0 legacy=0x7f55a8141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient: _finish_hunting 0 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 1 monclient: found mon.noname-d 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:54:16.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 1 -- 172.21.15.64:0/4023777184 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a809dd40 con 0x7f55a8135c80 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient: _finish_auth 0 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient: _check_auth_tickets 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 1 -- 172.21.15.64:0/4023777184 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5598001830 con 0x7f55a8135c80 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 5 monclient: authenticate success, global_id 174237 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient: dump: 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.714 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 1 -- 172.21.15.64:0/4023777184 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5598001a50 con 0x7f55a8135c80 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:16.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 1 -- 172.21.15.64:0/4023777184 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5598003950 con 0x7f55a8135c80 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55afe47640 4 set_mon_vals no callback set 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.703+0000 7f55acbe1640 10 monclient: dump: 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.704+0000 7f55afe47640 10 monclient: shutdown 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.704+0000 7f55afe47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.704+0000 7f55afe47640 1 -- 172.21.15.64:0/4023777184 >> v1:172.21.15.64:6792/0 conn(0x7f55a8135c80 legacy=0x7f55a813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.704+0000 7f55afe47640 1 -- 172.21.15.64:0/4023777184 shutdown_connections 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.704+0000 7f55afe47640 1 -- 172.21.15.64:0/4023777184 >> 172.21.15.64:0/4023777184 conn(0x7f55a806daa0 msgr2=0x7f55a80ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.704+0000 7f55afe47640 1 -- 172.21.15.64:0/4023777184 shutdown_connections 2024-09-17T15:54:16.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.704+0000 7f55afe47640 1 -- 172.21.15.64:0/4023777184 wait complete. 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.705+0000 7f55afe47640 10 monclient: build_initial_monmap 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.705+0000 7f55afe47640 10 monclient: monmap: 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:16.718 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:16.719 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:16.719 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:16.719 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:16.719 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.705+0000 7f55afe47640 1 Processor -- start 2024-09-17T15:54:16.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.705+0000 7f55afe47640 1 -- start start 2024-09-17T15:54:16.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.705+0000 7f55afe47640 10 monclient: init 2024-09-17T15:54:16.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.705+0000 7f55afe47640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:16.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.705+0000 7f55afe47640 10 monclient: _add_conns ranks=[2,0,1,6,8,7,5,3,4] 2024-09-17T15:54:16.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.705+0000 7f55afe47640 10 monclient(hunting): picked mon.noname-b con 0x7f55a80a54b0 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:16.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55afe47640 10 monclient(hunting): picked mon.noname-a con 0x7f55a8135c80 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:16.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55afe47640 10 monclient(hunting): picked mon.noname-f con 0x7f55a81360a0 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:16.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55afe47640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55a80a07f0 con 0x7f55a8135c80 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55afe47640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55a809f120 con 0x7f55a81360a0 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55afe47640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55a80a0f60 con 0x7f55a80a54b0 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55afe47640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55afe47640 10 monclient(hunting): authenticate will time out at 1254.381048s 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55adbe3640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f55a80a54b0 0x7f55a809e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53628/0 (socket says 172.21.15.64:53628) 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55adbe3640 1 -- 172.21.15.64:0/527604429 learned_addr learned my addr 172.21.15.64:0/527604429 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55ad3e2640 1 --1- 172.21.15.64:0/527604429 >> v1:172.21.15.64:6789/0 conn(0x7f55a8135c80 0x7f55a809ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35324/0 (socket says 172.21.15.64:35324) 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3068323917 0 0) 0x7f55a80a07f0 con 0x7f55a8135c80 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): my global_id is 125172 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f558c003520 con 0x7f55a8135c80 2024-09-17T15:54:16.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1893810206 0 0) 0x7f55a80a0f60 con 0x7f55a80a54b0 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): my global_id is 125300 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55a80a07f0 con 0x7f55a80a54b0 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2497295310 0 0) 0x7f55a809f120 con 0x7f55a81360a0 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): my global_id is 6967 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55a80a0f60 con 0x7f55a81360a0 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 917197369 0 0) 0x7f55a80a07f0 con 0x7f55a80a54b0 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55a809f120 con 0x7f55a80a54b0 2024-09-17T15:54:16.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f55a40014a0 con 0x7f55a80a54b0 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): dump: 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.722 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.723 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.723 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.723 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.723 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.723 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:16.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1066663002 0 0) 0x7f558c003520 con 0x7f55a8135c80 2024-09-17T15:54:16.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55a80a07f0 con 0x7f55a8135c80 2024-09-17T15:54:16.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f55980017f0 con 0x7f55a8135c80 2024-09-17T15:54:16.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:16.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:16.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): dump: 2024-09-17T15:54:16.723 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.724 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3256310994 0 0) 0x7f55a80a0f60 con 0x7f55a81360a0 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f558c003520 con 0x7f55a81360a0 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f55a0001c70 con 0x7f55a81360a0 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient(hunting): dump: 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1754718515 0 0) 0x7f55a809f120 con 0x7f55a80a54b0 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 >> v1:172.21.15.100:6789/0 conn(0x7f55a81360a0 legacy=0x7f55a81d6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 >> v1:172.21.15.64:6789/0 conn(0x7f55a8135c80 legacy=0x7f55a809ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:54:16.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 monclient: found mon.b 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a80a0ad0 con 0x7f55a80a54b0 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f559e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55afe47640 5 monclient: authenticate success, global_id 125300 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55afe47640 10 monclient: _renew_subs 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55afe47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55afe47640 1 -- 172.21.15.64:0/527604429 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f55a80a7340 con 0x7f55a80a54b0 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.706+0000 7f55afe47640 10 monclient: _renew_subs 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.707+0000 7f55afe47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.707+0000 7f55afe47640 1 -- 172.21.15.64:0/527604429 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f55a80a0510 con 0x7f55a80a54b0 2024-09-17T15:54:16.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.707+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f55a4004a90 con 0x7f55a80a54b0 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.707+0000 7f559e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.707+0000 7f559ffff640 4 set_mon_vals no callback set 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.707+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f55a4004f80 con 0x7f55a80a54b0 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.707+0000 7f559e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.707+0000 7f559e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.707+0000 7f559e7fc640 10 monclient: dump: 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:16.728 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.707+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f55a4011860 con 0x7f55a80a54b0 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.708+0000 7f55afe47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.708+0000 7f55afe47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.708+0000 7f55afe47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.708+0000 7f55afe47640 1 -- 172.21.15.64:0/527604429 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5590005080 con 0x7f55a80a54b0 2024-09-17T15:54:16.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.708+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f55a404e9c0 con 0x7f55a80a54b0 2024-09-17T15:54:16.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.708+0000 7f55ad3e2640 10 monclient: get_auth_request con 0x7f558c040b90 auth_method 0 2024-09-17T15:54:16.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.712+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f55a40176f0 con 0x7f55a80a54b0 2024-09-17T15:54:16.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.712+0000 7f559e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:16.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.712+0000 7f559e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:16.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.897+0000 7f55afe47640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:54:16.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.897+0000 7f55afe47640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:54:16.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.897+0000 7f55afe47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:16.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.897+0000 7f55afe47640 1 -- 172.21.15.64:0/527604429 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5590005370 con 0x7f55a80a54b0 2024-09-17T15:54:16.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.897+0000 7f559e7fc640 1 -- 172.21.15.64:0/527604429 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v327) ==== 74+0+10878 (unknown 3445726790 0 739351567) 0x7f55a401bad0 con 0x7f55a80a54b0 2024-09-17T15:54:16.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.897+0000 7f559e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:54:16.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.897+0000 7f559e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:16.900 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:54:16.900 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":327,"fsid":"3de99658-3189-432a-9efe-6c9d4c0a2934","created":"2024-09-17T15:45:43.591977+0000","modified":"2024-09-17T15:53:06.583627+0000","last_up_change":"2024-09-17T15:45:55.895333+0000","last_in_change":"2024-09-17T15:45:45.218655+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":100,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T15:45:53.848072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T15:46:00.391921+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1d9b48f9-b24c-453f-a5e9-79d74f56118c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":325,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":796819178}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":796819178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":796819178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":796819178}]},"public_addr":"172.21.15.64:6808/796819178","cluster_addr":"172.21.15.64:6809/796819178","heartbeat_back_addr":"172.21.15.64:6811/796819178","heartbeat_front_addr":"172.21.15.64:6810/796819178","state":["exists","up"]},{"osd":1,"uuid":"6b8dcb8b-604c-46d3-a682-5ef3e93155d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":325,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6800","nonce":3889666033}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":3889666033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":3889666033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":3889666033}]},"public_addr":"172.21.15.64:6800/3889666033","cluster_addr":"172.21.15.64:6801/3889666033","heartbeat_back_addr":"172.21.15.64:6803/3889666033","heartbeat_front_addr":"172.21.15.64:6802/3889666033","state":["exists","up"]},{"osd":2,"uuid":"34a266c9-43fe-4e7b-ae85-df0115c5f618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":325,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":1825798494}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":1825798494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":1825798494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":1825798494}]},"public_addr":"172.21.15.64:6804/1825798494","cluster_addr":"172.21.15.64:6805/1825798494","heartbeat_back_addr":"172.21.15.64:6807/1825798494","heartbeat_front_addr":"172.21.15.64:6806/1825798494","state":["exists","up"]},{"osd":3,"uuid":"c8740bab-d046-474f-b6a4-84333eb1e094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":325,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6804","nonce":2791828047}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6805","nonce":2791828047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6807","nonce":2791828047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6806","nonce":2791828047}]},"public_addr":"172.21.15.100:6804/2791828047","cluster_addr":"172.21.15.100:6805/2791828047","heartbeat_back_addr":"172.21.15.100:6807/2791828047","heartbeat_front_addr":"172.21.15.100:6806/2791828047","state":["exists","up"]},{"osd":4,"uuid":"08a1700b-63c5-4028-824f-dbd38931ddb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":325,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6808","nonce":4242749853}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6809","nonce":4242749853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6811","nonce":4242749853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6810","nonce":4242749853}]},"public_addr":"172.21.15.100:6808/4242749853","cluster_addr":"172.21.15.100:6809/4242749853","heartbeat_back_addr":"172.21.15.100:6811/4242749853","heartbeat_front_addr":"172.21.15.100:6810/4242749853","state":["exists","up"]},{"osd":5,"uuid":"81fa403f-39e9-44fd-8b1f-ef72fd7ba855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":325,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6800","nonce":4145568194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6801","nonce":4145568194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6803","nonce":4145568194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.100:6802","nonce":4145568194}]},"public_addr":"172.21.15.100:6800/4145568194","cluster_addr":"172.21.15.100:6801/4145568194","heartbeat_back_addr":"172.21.15.100:6803/4145568194","heartbeat_front_addr":"172.21.15.100:6802/4145568194","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.703397+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.212068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.432455+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.332164+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.533990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:45:51.237115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:54:16.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.900+0000 7f55afe47640 1 -- 172.21.15.64:0/527604429 >> v1:172.21.15.100:6812/4210963484 conn(0x7f558c040b90 legacy=0x7f558c043050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.900+0000 7f55afe47640 10 monclient: shutdown 2024-09-17T15:54:16.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.900+0000 7f55afe47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:16.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.900+0000 7f55afe47640 1 -- 172.21.15.64:0/527604429 >> v1:172.21.15.64:6790/0 conn(0x7f55a80a54b0 legacy=0x7f55a809e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:16.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.900+0000 7f55afe47640 1 -- 172.21.15.64:0/527604429 shutdown_connections 2024-09-17T15:54:16.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.900+0000 7f55afe47640 1 -- 172.21.15.64:0/527604429 >> 172.21.15.64:0/527604429 conn(0x7f55a806daa0 msgr2=0x7f55a8133ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:16.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.900+0000 7f55afe47640 1 -- 172.21.15.64:0/527604429 shutdown_connections 2024-09-17T15:54:16.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:16.900+0000 7f55afe47640 1 -- 172.21.15.64:0/527604429 wait complete. 2024-09-17T15:54:17.913 INFO:tasks.ceph:Scrubbing osd.0 2024-09-17T15:54:17.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-17T15:54:18.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.030+0000 7f6d57d82640 10 monclient: get_monmap_and_config 2024-09-17T15:54:18.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.030+0000 7f6d57d82640 10 monclient: build_initial_monmap 2024-09-17T15:54:18.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.030+0000 7f6d57d82640 10 monclient: monmap: 2024-09-17T15:54:18.032 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:18.031658+0000 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:18.031658+0000 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:18.033 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:18.034 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.031+0000 7f6d57d82640 1 Processor -- start 2024-09-17T15:54:18.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.031+0000 7f6d57d82640 1 -- start start 2024-09-17T15:54:18.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.031+0000 7f6d57d82640 10 monclient: init 2024-09-17T15:54:18.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d57d82640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:18.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d57d82640 10 monclient: _add_conns ranks=[3,5,2,4,0,7,1,8,6] 2024-09-17T15:54:18.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d57d82640 10 monclient(hunting): picked mon.noname-g con 0x7f6d50138d80 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:18.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d57d82640 10 monclient(hunting): picked mon.noname-h con 0x7f6d500a5de0 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:18.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d57d82640 10 monclient(hunting): picked mon.noname-b con 0x7f6d5009cd20 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:18.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d57d82640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d500a8390 con 0x7f6d5009cd20 2024-09-17T15:54:18.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d57d82640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d5009db80 con 0x7f6d50138d80 2024-09-17T15:54:18.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d57d82640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d5009de60 con 0x7f6d500a5de0 2024-09-17T15:54:18.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d57d82640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:18.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d57d82640 10 monclient(hunting): authenticate will time out at 1255.707261s 2024-09-17T15:54:18.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d5631f640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f6d5009cd20 0x7f6d50141830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53638/0 (socket says 172.21.15.64:53638) 2024-09-17T15:54:18.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d5631f640 1 -- 172.21.15.64:0/754015265 learned_addr learned my addr 172.21.15.64:0/754015265 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:18.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 1 -- 172.21.15.64:0/754015265 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2611226431 0 0) 0x7f6d500a8390 con 0x7f6d5009cd20 2024-09-17T15:54:18.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient(hunting): my global_id is 125309 2024-09-17T15:54:18.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 1 -- 172.21.15.64:0/754015265 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d3c003520 con 0x7f6d5009cd20 2024-09-17T15:54:18.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 1 -- 172.21.15.64:0/754015265 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 997168073 0 0) 0x7f6d5009db80 con 0x7f6d50138d80 2024-09-17T15:54:18.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient(hunting): my global_id is 115968 2024-09-17T15:54:18.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 1 -- 172.21.15.64:0/754015265 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d500a8390 con 0x7f6d50138d80 2024-09-17T15:54:18.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 1 -- 172.21.15.64:0/754015265 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 368716977 0 0) 0x7f6d5009de60 con 0x7f6d500a5de0 2024-09-17T15:54:18.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient(hunting): my global_id is 174290 2024-09-17T15:54:18.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 1 -- 172.21.15.64:0/754015265 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d5009db80 con 0x7f6d500a5de0 2024-09-17T15:54:18.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 1 -- 172.21.15.64:0/754015265 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 135640437 0 0) 0x7f6d3c003520 con 0x7f6d5009cd20 2024-09-17T15:54:18.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 1 -- 172.21.15.64:0/754015265 >> v1:172.21.15.100:6791/0 conn(0x7f6d500a5de0 legacy=0x7f6d5013e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 1 -- 172.21.15.64:0/754015265 >> v1:172.21.15.100:6790/0 conn(0x7f6d50138d80 legacy=0x7f6d500a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient: _finish_hunting 0 2024-09-17T15:54:18.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 1 monclient: found mon.noname-b 2024-09-17T15:54:18.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:18.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:54:18.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 1 -- 172.21.15.64:0/754015265 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d5009e140 con 0x7f6d5009cd20 2024-09-17T15:54:18.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient: _finish_auth 0 2024-09-17T15:54:18.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient: _check_auth_tickets 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 1 -- 172.21.15.64:0/754015265 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6d48001bb0 con 0x7f6d5009cd20 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d57d82640 5 monclient: authenticate success, global_id 125309 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d57d82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.032+0000 7f6d54b1c640 10 monclient: dump: 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.045 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d57d82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d54b1c640 1 -- 172.21.15.64:0/754015265 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6d48001d50 con 0x7f6d5009cd20 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d54b1c640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d54b1c640 1 -- 172.21.15.64:0/754015265 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6d48004f60 con 0x7f6d5009cd20 2024-09-17T15:54:18.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d54b1c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d54b1c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d54b1c640 10 monclient: dump: 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.047 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.048 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d57d82640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d57d82640 4 set_mon_vals no callback set 2024-09-17T15:54:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d57d82640 10 monclient: shutdown 2024-09-17T15:54:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d57d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d57d82640 1 -- 172.21.15.64:0/754015265 >> v1:172.21.15.64:6790/0 conn(0x7f6d5009cd20 legacy=0x7f6d50141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d57d82640 1 -- 172.21.15.64:0/754015265 shutdown_connections 2024-09-17T15:54:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d57d82640 1 -- 172.21.15.64:0/754015265 >> 172.21.15.64:0/754015265 conn(0x7f6d5006dc70 msgr2=0x7f6d500abb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d57d82640 1 -- 172.21.15.64:0/754015265 shutdown_connections 2024-09-17T15:54:18.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.033+0000 7f6d57d82640 1 -- 172.21.15.64:0/754015265 wait complete. 2024-09-17T15:54:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.034+0000 7f6d57d82640 10 monclient: build_initial_monmap 2024-09-17T15:54:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.034+0000 7f6d57d82640 10 monclient: monmap: 2024-09-17T15:54:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:18.049 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:18.050 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.034+0000 7f6d57d82640 1 Processor -- start 2024-09-17T15:54:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.034+0000 7f6d57d82640 1 -- start start 2024-09-17T15:54:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.034+0000 7f6d57d82640 10 monclient: init 2024-09-17T15:54:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d57d82640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d57d82640 10 monclient: _add_conns ranks=[7,0,3,2,5,1,6,4,8] 2024-09-17T15:54:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d57d82640 10 monclient(hunting): picked mon.noname-i con 0x7f6d5009cd20 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d57d82640 10 monclient(hunting): picked mon.noname-a con 0x7f6d500a5de0 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:18.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d57d82640 10 monclient(hunting): picked mon.noname-g con 0x7f6d50138d80 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d57d82640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d5013b140 con 0x7f6d500a5de0 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d57d82640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d501cede0 con 0x7f6d50138d80 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d57d82640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d500a4f60 con 0x7f6d5009cd20 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d57d82640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d57d82640 10 monclient(hunting): authenticate will time out at 1255.710399s 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d5531d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f6d500a5de0 0x7f6d501ce6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35332/0 (socket says 172.21.15.64:35332) 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d5531d640 1 -- 172.21.15.64:0/3590214384 learned_addr learned my addr 172.21.15.64:0/3590214384 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1141636510 0 0) 0x7f6d5013b140 con 0x7f6d500a5de0 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 10 monclient(hunting): my global_id is 125181 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d34003520 con 0x7f6d500a5de0 2024-09-17T15:54:18.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 169228472 0 0) 0x7f6d501cede0 con 0x7f6d50138d80 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 10 monclient(hunting): my global_id is 115977 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d5013b140 con 0x7f6d50138d80 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1175010407 0 0) 0x7f6d500a4f60 con 0x7f6d5009cd20 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 10 monclient(hunting): my global_id is 174211 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d501cede0 con 0x7f6d5009cd20 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 691378557 0 0) 0x7f6d34003520 con 0x7f6d500a5de0 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d500a4f60 con 0x7f6d500a5de0 2024-09-17T15:54:18.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6d40001b70 con 0x7f6d500a5de0 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.035+0000 7f6d467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient(hunting): dump: 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.053 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.054 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3147147694 0 0) 0x7f6d5013b140 con 0x7f6d50138d80 2024-09-17T15:54:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d34003520 con 0x7f6d50138d80 2024-09-17T15:54:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6d48001a70 con 0x7f6d50138d80 2024-09-17T15:54:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-17T15:54:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient(hunting): dump: 2024-09-17T15:54:18.054 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.055 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.056 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3256787578 0 0) 0x7f6d501cede0 con 0x7f6d5009cd20 2024-09-17T15:54:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d5013b140 con 0x7f6d5009cd20 2024-09-17T15:54:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6d4c0019f0 con 0x7f6d5009cd20 2024-09-17T15:54:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:54:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient(hunting): dump: 2024-09-17T15:54:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.056 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.057 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2901482402 0 0) 0x7f6d500a4f60 con 0x7f6d500a5de0 2024-09-17T15:54:18.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 >> v1:172.21.15.100:6792/0 conn(0x7f6d5009cd20 legacy=0x7f6d501cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 >> v1:172.21.15.100:6790/0 conn(0x7f6d50138d80 legacy=0x7f6d501d6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient: _finish_hunting 0 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 1 monclient: found mon.a 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d500aa500 con 0x7f6d500a5de0 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient: _finish_auth 0 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient: _check_auth_tickets 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d57d82640 5 monclient: authenticate success, global_id 125181 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d57d82640 10 monclient: _renew_subs 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d57d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d57d82640 1 -- 172.21.15.64:0/3590214384 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6d500a9fb0 con 0x7f6d500a5de0 2024-09-17T15:54:18.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d57d82640 10 monclient: _renew_subs 2024-09-17T15:54:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d57d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d57d82640 1 -- 172.21.15.64:0/3590214384 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6d5013ae60 con 0x7f6d500a5de0 2024-09-17T15:54:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6d400038c0 con 0x7f6d500a5de0 2024-09-17T15:54:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f6d40003e30 con 0x7f6d500a5de0 2024-09-17T15:54:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d47fff640 4 set_mon_vals no callback set 2024-09-17T15:54:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.036+0000 7f6d467fc640 10 monclient: dump: 2024-09-17T15:54:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.059 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.060 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.060 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.037+0000 7f6d57d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.037+0000 7f6d57d82640 1 -- 172.21.15.64:0/3590214384 --> v1:172.21.15.64:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f6d38000f80 con 0x7f6d500a5de0 2024-09-17T15:54:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.037+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f6d40003fd0 con 0x7f6d500a5de0 2024-09-17T15:54:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.037+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f6d4004eaa0 con 0x7f6d500a5de0 2024-09-17T15:54:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.037+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 --> v1:172.21.15.64:6808/796819178 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d3404b570 con 0x7f6d34047900 2024-09-17T15:54:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.038+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_get_version_reply(handle=1 version=327) ==== 24+0+0 (unknown 900077626 0 0) 0x7f6d4004ef10 con 0x7f6d500a5de0 2024-09-17T15:54:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.038+0000 7f6d467fc640 10 monclient: handle_get_version_reply finishing 1 version 327 2024-09-17T15:54:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.038+0000 7f6d55b1e640 10 monclient: get_auth_request con 0x7f6d34040d30 auth_method 0 2024-09-17T15:54:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.038+0000 7f6d5631f640 10 monclient: get_auth_request con 0x7f6d34047900 auth_method 0 2024-09-17T15:54:18.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.038+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== osd.0 v1:172.21.15.64:6808/796819178 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d3404b570 con 0x7f6d34047900 2024-09-17T15:54:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.051+0000 7f6d57d82640 1 -- 172.21.15.64:0/3590214384 --> v1:172.21.15.64:6808/796819178 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6d38002cd0 con 0x7f6d34047900 2024-09-17T15:54:18.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.051+0000 7f6d467fc640 1 -- 172.21.15.64:0/3590214384 <== osd.0 v1:172.21.15.64:6808/796819178 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f6d38002cd0 con 0x7f6d34047900 2024-09-17T15:54:18.062 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:18.062 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T15:54:18.062 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.052+0000 7f6d57d82640 1 -- 172.21.15.64:0/3590214384 >> v1:172.21.15.64:6808/796819178 conn(0x7f6d34047900 legacy=0x7f6d34049d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.052+0000 7f6d57d82640 1 -- 172.21.15.64:0/3590214384 >> v1:172.21.15.100:6812/4210963484 conn(0x7f6d34040d30 legacy=0x7f6d340431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.052+0000 7f6d57d82640 10 monclient: shutdown 2024-09-17T15:54:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.052+0000 7f6d57d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.052+0000 7f6d57d82640 1 -- 172.21.15.64:0/3590214384 >> v1:172.21.15.64:6789/0 conn(0x7f6d500a5de0 legacy=0x7f6d501ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.052+0000 7f6d5631f640 1 -- 172.21.15.64:0/3590214384 reap_dead start 2024-09-17T15:54:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.052+0000 7f6d57d82640 1 -- 172.21.15.64:0/3590214384 shutdown_connections 2024-09-17T15:54:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.052+0000 7f6d57d82640 1 -- 172.21.15.64:0/3590214384 >> 172.21.15.64:0/3590214384 conn(0x7f6d5006dc70 msgr2=0x7f6d50134fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.052+0000 7f6d57d82640 1 -- 172.21.15.64:0/3590214384 shutdown_connections 2024-09-17T15:54:18.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.052+0000 7f6d57d82640 1 -- 172.21.15.64:0/3590214384 wait complete. 2024-09-17T15:54:18.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-17T15:54:18.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.166+0000 7f0f9637d640 10 monclient: get_monmap_and_config 2024-09-17T15:54:18.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.166+0000 7f0f9637d640 10 monclient: build_initial_monmap 2024-09-17T15:54:18.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.166+0000 7f0f9637d640 10 monclient: monmap: 2024-09-17T15:54:18.168 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:18.168 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.168 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:18.167043+0000 2024-09-17T15:54:18.168 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:18.167043+0000 2024-09-17T15:54:18.168 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:18.168 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.168 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:18.168 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:18.168 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:18.169 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:18.169 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:18.169 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:18.169 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:18.169 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:18.169 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:18.169 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.166+0000 7f0f9637d640 1 Processor -- start 2024-09-17T15:54:18.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.166+0000 7f0f9637d640 1 -- start start 2024-09-17T15:54:18.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.166+0000 7f0f9637d640 10 monclient: init 2024-09-17T15:54:18.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f9637d640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:18.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f9637d640 10 monclient: _add_conns ranks=[4,8,6,0,2,3,5,1,7] 2024-09-17T15:54:18.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f9637d640 10 monclient(hunting): picked mon.noname-c con 0x7f0f900a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:18.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f9637d640 10 monclient(hunting): picked mon.noname-e con 0x7f0f90135b50 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:18.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f9637d640 10 monclient(hunting): picked mon.noname-d con 0x7f0f90135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:18.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f9637d640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f9009d6f0 con 0x7f0f900a5520 2024-09-17T15:54:18.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f9637d640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f9009d9d0 con 0x7f0f90135f70 2024-09-17T15:54:18.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f9637d640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f9009dcb0 con 0x7f0f90135b50 2024-09-17T15:54:18.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f9637d640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:18.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f9637d640 10 monclient(hunting): authenticate will time out at 1255.842542s 2024-09-17T15:54:18.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8f7fe640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f0f90135b50 0x7f0f9013deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51100/0 (socket says 172.21.15.64:51100) 2024-09-17T15:54:18.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8ffff640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f0f900a5520 0x7f0f900a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:59842/0 (socket says 172.21.15.64:59842) 2024-09-17T15:54:18.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8f7fe640 1 -- 172.21.15.64:0/2092933583 learned_addr learned my addr 172.21.15.64:0/2092933583 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:18.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 1 -- 172.21.15.64:0/2092933583 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2587974084 0 0) 0x7f0f9009dcb0 con 0x7f0f90135b50 2024-09-17T15:54:18.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 10 monclient(hunting): my global_id is 174221 2024-09-17T15:54:18.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 1 -- 172.21.15.64:0/2092933583 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f78003520 con 0x7f0f90135b50 2024-09-17T15:54:18.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 1 -- 172.21.15.64:0/2092933583 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3754341193 0 0) 0x7f0f9009d9d0 con 0x7f0f90135f70 2024-09-17T15:54:18.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 10 monclient(hunting): my global_id is 174246 2024-09-17T15:54:18.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 1 -- 172.21.15.64:0/2092933583 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f9009dcb0 con 0x7f0f90135f70 2024-09-17T15:54:18.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 1 -- 172.21.15.64:0/2092933583 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4214917657 0 0) 0x7f0f9009d6f0 con 0x7f0f900a5520 2024-09-17T15:54:18.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.167+0000 7f0f8effd640 10 monclient(hunting): my global_id is 154723 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 1 -- 172.21.15.64:0/2092933583 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f9009d9d0 con 0x7f0f900a5520 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 1 -- 172.21.15.64:0/2092933583 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3271169529 0 0) 0x7f0f9009dcb0 con 0x7f0f90135f70 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 1 -- 172.21.15.64:0/2092933583 >> v1:172.21.15.64:6793/0 conn(0x7f0f90135b50 legacy=0x7f0f9013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 1 -- 172.21.15.64:0/2092933583 >> v1:172.21.15.64:6791/0 conn(0x7f0f900a5520 legacy=0x7f0f900a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 10 monclient: _finish_hunting 0 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 1 monclient: found mon.noname-d 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 1 -- 172.21.15.64:0/2092933583 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f9009ccd0 con 0x7f0f90135f70 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 10 monclient: _finish_auth 0 2024-09-17T15:54:18.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 10 monclient: _check_auth_tickets 2024-09-17T15:54:18.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:18.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 1 -- 172.21.15.64:0/2092933583 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0f88001ac0 con 0x7f0f90135f70 2024-09-17T15:54:18.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f9637d640 5 monclient: authenticate success, global_id 174246 2024-09-17T15:54:18.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f9637d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:18.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:54:18.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 10 monclient: dump: 2024-09-17T15:54:18.180 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.180 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.180 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.180 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.180 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.180 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f9637d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 1 -- 172.21.15.64:0/2092933583 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0f88001c60 con 0x7f0f90135f70 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:18.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 1 -- 172.21.15.64:0/2092933583 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0f88003cd0 con 0x7f0f90135f70 2024-09-17T15:54:18.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f9637d640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:18.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f9637d640 4 set_mon_vals no callback set 2024-09-17T15:54:18.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:18.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f8effd640 10 monclient: dump: 2024-09-17T15:54:18.182 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.182 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.182 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.182 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.182 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.182 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.182 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.182 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f9637d640 10 monclient: shutdown 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f9637d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f9637d640 1 -- 172.21.15.64:0/2092933583 >> v1:172.21.15.64:6792/0 conn(0x7f0f90135f70 legacy=0x7f0f901415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f9637d640 1 -- 172.21.15.64:0/2092933583 shutdown_connections 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f9637d640 1 -- 172.21.15.64:0/2092933583 >> 172.21.15.64:0/2092933583 conn(0x7f0f9006db30 msgr2=0x7f0f900ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:18.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f9637d640 1 -- 172.21.15.64:0/2092933583 shutdown_connections 2024-09-17T15:54:18.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.168+0000 7f0f9637d640 1 -- 172.21.15.64:0/2092933583 wait complete. 2024-09-17T15:54:18.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.169+0000 7f0f9637d640 10 monclient: build_initial_monmap 2024-09-17T15:54:18.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.169+0000 7f0f9637d640 10 monclient: monmap: 2024-09-17T15:54:18.184 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:18.184 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:18.184 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:18.184 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:18.184 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:18.184 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:18.184 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:18.184 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:18.184 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:18.184 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:18.185 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.169+0000 7f0f9637d640 1 Processor -- start 2024-09-17T15:54:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9637d640 1 -- start start 2024-09-17T15:54:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9637d640 10 monclient: init 2024-09-17T15:54:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9637d640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9637d640 10 monclient: _add_conns ranks=[1,7,4,5,6,3,2,0,8] 2024-09-17T15:54:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9637d640 10 monclient(hunting): picked mon.noname-f con 0x7f0f900a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9637d640 10 monclient(hunting): picked mon.noname-i con 0x7f0f90135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9637d640 10 monclient(hunting): picked mon.noname-c con 0x7f0f90135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9637d640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f900a8f70 con 0x7f0f900a5520 2024-09-17T15:54:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9637d640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f90135310 con 0x7f0f90135f70 2024-09-17T15:54:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9637d640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f90134e00 con 0x7f0f90135b50 2024-09-17T15:54:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9637d640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9637d640 10 monclient(hunting): authenticate will time out at 1255.845491s 2024-09-17T15:54:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9491a640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f0f90135f70 0x7f0f901d5880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:59850/0 (socket says 172.21.15.64:59850) 2024-09-17T15:54:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f9491a640 1 -- 172.21.15.64:0/1582793826 learned_addr learned my addr 172.21.15.64:0/1582793826 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2245835042 0 0) 0x7f0f90135310 con 0x7f0f90135f70 2024-09-17T15:54:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f8cff9640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f8cff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f8cff9640 10 monclient(hunting): my global_id is 154732 2024-09-17T15:54:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f7c003550 con 0x7f0f90135f70 2024-09-17T15:54:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4264351936 0 0) 0x7f0f90134e00 con 0x7f0f90135b50 2024-09-17T15:54:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f8cff9640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f8cff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f8cff9640 10 monclient(hunting): my global_id is 174220 2024-09-17T15:54:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f90135310 con 0x7f0f90135b50 2024-09-17T15:54:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.170+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 534371507 0 0) 0x7f0f7c003550 con 0x7f0f90135f70 2024-09-17T15:54:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f90134e00 con 0x7f0f90135f70 2024-09-17T15:54:18.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0f88003970 con 0x7f0f90135f70 2024-09-17T15:54:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient(hunting): dump: 2024-09-17T15:54:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.189 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.189 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.189 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.189 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.192 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.193 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:18.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4199174324 0 0) 0x7f0f80001a20 con 0x7f0f900a5520 2024-09-17T15:54:18.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient(hunting): my global_id is 6976 2024-09-17T15:54:18.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f7c003550 con 0x7f0f900a5520 2024-09-17T15:54:18.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4112012517 0 0) 0x7f0f90135310 con 0x7f0f90135b50 2024-09-17T15:54:18.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f80001a20 con 0x7f0f90135b50 2024-09-17T15:54:18.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2667066491 0 0) 0x7f0f90134e00 con 0x7f0f90135f70 2024-09-17T15:54:18.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 >> v1:172.21.15.100:6792/0 conn(0x7f0f90135b50 legacy=0x7f0f900a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 >> v1:172.21.15.100:6789/0 conn(0x7f0f900a5520 legacy=0x7f0f901cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient: _finish_hunting 0 2024-09-17T15:54:18.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 1 monclient: found mon.c 2024-09-17T15:54:18.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:18.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:18.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f90132fa0 con 0x7f0f90135f70 2024-09-17T15:54:18.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient: _finish_auth 0 2024-09-17T15:54:18.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient: _check_auth_tickets 2024-09-17T15:54:18.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:18.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f9637d640 5 monclient: authenticate success, global_id 154732 2024-09-17T15:54:18.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f9637d640 10 monclient: _renew_subs 2024-09-17T15:54:18.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f9637d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:18.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f9637d640 1 -- 172.21.15.64:0/1582793826 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0f900a0100 con 0x7f0f90135f70 2024-09-17T15:54:18.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f9637d640 10 monclient: _renew_subs 2024-09-17T15:54:18.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f9637d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:18.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f9637d640 1 -- 172.21.15.64:0/1582793826 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f0f900a5050 con 0x7f0f90135f70 2024-09-17T15:54:18.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0f88001c40 con 0x7f0f90135f70 2024-09-17T15:54:18.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:18.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f0f88004100 con 0x7f0f90135f70 2024-09-17T15:54:18.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8e7fc640 4 set_mon_vals no callback set 2024-09-17T15:54:18.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:18.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.171+0000 7f0f8cff9640 10 monclient: dump: 2024-09-17T15:54:18.195 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.196 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.197 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.172+0000 7f0f9637d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.172+0000 7f0f9637d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.172+0000 7f0f9637d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.172+0000 7f0f9637d640 1 -- 172.21.15.64:0/1582793826 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f70005080 con 0x7f0f90135f70 2024-09-17T15:54:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.172+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f0f880042a0 con 0x7f0f90135f70 2024-09-17T15:54:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.172+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f0f8804ecb0 con 0x7f0f90135f70 2024-09-17T15:54:18.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.172+0000 7f0f8ffff640 10 monclient: get_auth_request con 0x7f0f7c040620 auth_method 0 2024-09-17T15:54:18.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.177+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f0f88017a20 con 0x7f0f90135f70 2024-09-17T15:54:18.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.177+0000 7f0f8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:18.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.177+0000 7f0f8cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.359+0000 7f0f9637d640 1 -- 172.21.15.64:0/1582793826 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f0f70000b90 con 0x7f0f7c040620 2024-09-17T15:54:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.360+0000 7f0f8cff9640 1 -- 172.21.15.64:0/1582793826 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (unknown 2057319476 0 0) 0x7f0f70000b90 con 0x7f0f7c040620 2024-09-17T15:54:18.362 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 0 to deep-scrub 2024-09-17T15:54:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.362+0000 7f0f9637d640 1 -- 172.21.15.64:0/1582793826 >> v1:172.21.15.100:6812/4210963484 conn(0x7f0f7c040620 legacy=0x7f0f7c042ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.362+0000 7f0f9637d640 10 monclient: shutdown 2024-09-17T15:54:18.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.362+0000 7f0f9637d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.362+0000 7f0f9637d640 1 -- 172.21.15.64:0/1582793826 >> v1:172.21.15.64:6791/0 conn(0x7f0f90135f70 legacy=0x7f0f901d5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.362+0000 7f0f9637d640 1 -- 172.21.15.64:0/1582793826 shutdown_connections 2024-09-17T15:54:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.362+0000 7f0f9637d640 1 -- 172.21.15.64:0/1582793826 >> 172.21.15.64:0/1582793826 conn(0x7f0f9006db30 msgr2=0x7f0f901334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.362+0000 7f0f9637d640 1 -- 172.21.15.64:0/1582793826 shutdown_connections 2024-09-17T15:54:18.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.362+0000 7f0f9637d640 1 -- 172.21.15.64:0/1582793826 wait complete. 2024-09-17T15:54:18.375 INFO:tasks.ceph:Scrubbing osd.1 2024-09-17T15:54:18.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-17T15:54:18.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.484+0000 7fc92f76b640 10 monclient: get_monmap_and_config 2024-09-17T15:54:18.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.484+0000 7fc92f76b640 10 monclient: build_initial_monmap 2024-09-17T15:54:18.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.484+0000 7fc92f76b640 10 monclient: monmap: 2024-09-17T15:54:18.486 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:18.486 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.486 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:18.485290+0000 2024-09-17T15:54:18.486 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:18.485290+0000 2024-09-17T15:54:18.486 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:18.486 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.487 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:18.487 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:18.487 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:18.487 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:18.487 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:18.487 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:18.487 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:18.487 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:18.487 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:18.487 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92f76b640 1 Processor -- start 2024-09-17T15:54:18.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92f76b640 1 -- start start 2024-09-17T15:54:18.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92f76b640 10 monclient: init 2024-09-17T15:54:18.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92f76b640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:18.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92f76b640 10 monclient: _add_conns ranks=[2,7,1,4,5,8,3,6,0] 2024-09-17T15:54:18.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92f76b640 10 monclient(hunting): picked mon.noname-b con 0x7fc928138d80 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:18.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92f76b640 10 monclient(hunting): picked mon.noname-i con 0x7fc9280a5de0 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:18.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92f76b640 10 monclient(hunting): picked mon.noname-f con 0x7fc92809cd20 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:18.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92f76b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9280a8390 con 0x7fc92809cd20 2024-09-17T15:54:18.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92f76b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92809db80 con 0x7fc928138d80 2024-09-17T15:54:18.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92f76b640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92809de60 con 0x7fc9280a5de0 2024-09-17T15:54:18.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92f76b640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:18.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92f76b640 10 monclient(hunting): authenticate will time out at 1256.160786s 2024-09-17T15:54:18.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92d507640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc928138d80 0x7fc9280a56d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53644/0 (socket says 172.21.15.64:53644) 2024-09-17T15:54:18.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.485+0000 7fc92d507640 1 -- 172.21.15.64:0/1148230844 learned_addr learned my addr 172.21.15.64:0/1148230844 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:18.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 1 -- 172.21.15.64:0/1148230844 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3070921843 0 0) 0x7fc92809db80 con 0x7fc928138d80 2024-09-17T15:54:18.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient(hunting): my global_id is 125318 2024-09-17T15:54:18.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 1 -- 172.21.15.64:0/1148230844 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc914003520 con 0x7fc928138d80 2024-09-17T15:54:18.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 1 -- 172.21.15.64:0/1148230844 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4141204086 0 0) 0x7fc914003520 con 0x7fc928138d80 2024-09-17T15:54:18.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 1 -- 172.21.15.64:0/1148230844 >> v1:172.21.15.100:6792/0 conn(0x7fc9280a5de0 legacy=0x7fc92813e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 1 -- 172.21.15.64:0/1148230844 >> v1:172.21.15.100:6789/0 conn(0x7fc92809cd20 legacy=0x7fc928141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:54:18.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 1 monclient: found mon.noname-b 2024-09-17T15:54:18.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:18.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:54:18.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 1 -- 172.21.15.64:0/1148230844 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92809e140 con 0x7fc928138d80 2024-09-17T15:54:18.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient: _finish_auth 0 2024-09-17T15:54:18.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:54:18.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:18.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 1 -- 172.21.15.64:0/1148230844 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc918001830 con 0x7fc928138d80 2024-09-17T15:54:18.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:54:18.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient: dump: 2024-09-17T15:54:18.497 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.498 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.498 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.498 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.498 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.498 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.498 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.498 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.498 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.498 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.498 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.498 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.498 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.498 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.499 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.499 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc92f76b640 5 monclient: authenticate success, global_id 125318 2024-09-17T15:54:18.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc92f76b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:18.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 1 -- 172.21.15.64:0/1148230844 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc918001a50 con 0x7fc928138d80 2024-09-17T15:54:18.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:18.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 1 -- 172.21.15.64:0/1148230844 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc918004be0 con 0x7fc928138d80 2024-09-17T15:54:18.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:18.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc91ffff640 10 monclient: dump: 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.501 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.501 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc92f76b640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:18.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc92f76b640 4 set_mon_vals no callback set 2024-09-17T15:54:18.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc92f76b640 10 monclient: shutdown 2024-09-17T15:54:18.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc92f76b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:18.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.486+0000 7fc92f76b640 1 -- 172.21.15.64:0/1148230844 >> v1:172.21.15.64:6790/0 conn(0x7fc928138d80 legacy=0x7fc9280a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.487+0000 7fc92f76b640 1 -- 172.21.15.64:0/1148230844 shutdown_connections 2024-09-17T15:54:18.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.487+0000 7fc92f76b640 1 -- 172.21.15.64:0/1148230844 >> 172.21.15.64:0/1148230844 conn(0x7fc92806dc70 msgr2=0x7fc9280abb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:18.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.487+0000 7fc92f76b640 1 -- 172.21.15.64:0/1148230844 shutdown_connections 2024-09-17T15:54:18.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.487+0000 7fc92f76b640 1 -- 172.21.15.64:0/1148230844 wait complete. 2024-09-17T15:54:18.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.487+0000 7fc92f76b640 10 monclient: build_initial_monmap 2024-09-17T15:54:18.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.487+0000 7fc92f76b640 10 monclient: monmap: 2024-09-17T15:54:18.502 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:18.502 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:18.502 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:18.502 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:18.502 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:18.502 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:18.502 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:18.502 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:18.502 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:18.502 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:18.503 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:18.503 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:18.503 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:18.503 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:18.503 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:18.503 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.487+0000 7fc92f76b640 1 Processor -- start 2024-09-17T15:54:18.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92f76b640 1 -- start start 2024-09-17T15:54:18.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92f76b640 10 monclient: init 2024-09-17T15:54:18.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92f76b640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:18.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92f76b640 10 monclient: _add_conns ranks=[3,7,5,1,4,2,0,8,6] 2024-09-17T15:54:18.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92f76b640 10 monclient(hunting): picked mon.noname-g con 0x7fc92809cd20 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:18.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92f76b640 10 monclient(hunting): picked mon.noname-i con 0x7fc9280a5de0 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:18.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92f76b640 10 monclient(hunting): picked mon.noname-h con 0x7fc928138d80 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:18.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92f76b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92813adf0 con 0x7fc92809cd20 2024-09-17T15:54:18.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92f76b640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92813b0d0 con 0x7fc928138d80 2024-09-17T15:54:18.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92f76b640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9280a9ec0 con 0x7fc9280a5de0 2024-09-17T15:54:18.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92f76b640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:18.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92f76b640 10 monclient(hunting): authenticate will time out at 1256.163502s 2024-09-17T15:54:18.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92d507640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fc92809cd20 0x7fc9280a34f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:57830/0 (socket says 172.21.15.64:57830) 2024-09-17T15:54:18.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92cd06640 1 --1- >> v1:172.21.15.100:6792/0 conn(0x7fc9280a5de0 0x7fc9280a3c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6792/0 says I am v1:172.21.15.64:54316/0 (socket says 172.21.15.64:54316) 2024-09-17T15:54:18.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc92d507640 1 -- 172.21.15.64:0/3319274594 learned_addr learned my addr 172.21.15.64:0/3319274594 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:18.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1881700804 0 0) 0x7fc9280a9ec0 con 0x7fc9280a5de0 2024-09-17T15:54:18.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 10 monclient(hunting): my global_id is 174238 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc90c003550 con 0x7fc9280a5de0 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1315591557 0 0) 0x7fc92813adf0 con 0x7fc92809cd20 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 10 monclient(hunting): my global_id is 115986 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9280a9ec0 con 0x7fc92809cd20 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1487361292 0 0) 0x7fc92813b0d0 con 0x7fc928138d80 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 10 monclient(hunting): my global_id is 174299 2024-09-17T15:54:18.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.488+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc92813adf0 con 0x7fc928138d80 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.3 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2485258031 0 0) 0x7fc9280a9ec0 con 0x7fc92809cd20 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc92813b0d0 con 0x7fc92809cd20 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.3 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc918001c00 con 0x7fc92809cd20 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient(hunting): dump: 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.506 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1699243824 0 0) 0x7fc90c003550 con 0x7fc9280a5de0 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9280a9ec0 con 0x7fc9280a5de0 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4048632503 0 0) 0x7fc92813adf0 con 0x7fc928138d80 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc90c003550 con 0x7fc928138d80 2024-09-17T15:54:18.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc924001a30 con 0x7fc9280a5de0 2024-09-17T15:54:18.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:18.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:54:18.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient(hunting): dump: 2024-09-17T15:54:18.508 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.508 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.508 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.508 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.508 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.508 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.508 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.508 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.508 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.508 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc920001c50 con 0x7fc928138d80 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient(hunting): dump: 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.509 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.3 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1054657711 0 0) 0x7fc92813b0d0 con 0x7fc92809cd20 2024-09-17T15:54:18.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 >> v1:172.21.15.100:6792/0 conn(0x7fc9280a5de0 legacy=0x7fc9280a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 >> v1:172.21.15.100:6791/0 conn(0x7fc928138d80 legacy=0x7fc9281d6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient: _finish_hunting 0 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 monclient: found mon.g 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9280a5a60 con 0x7fc92809cd20 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient: _finish_auth 0 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc92f76b640 5 monclient: authenticate success, global_id 115986 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc92f76b640 10 monclient: _renew_subs 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc92f76b640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc92f76b640 1 -- 172.21.15.64:0/3319274594 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc92813ab10 con 0x7fc92809cd20 2024-09-17T15:54:18.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.3 v1:172.21.15.100:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc918004dd0 con 0x7fc92809cd20 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.3 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fc9180052c0 con 0x7fc92809cd20 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91dffb640 10 monclient: dump: 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.512 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc91f7fe640 4 set_mon_vals no callback set 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc92f76b640 10 monclient: _renew_subs 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc92f76b640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.489+0000 7fc92f76b640 1 -- 172.21.15.64:0/3319274594 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9280a44f0 con 0x7fc92809cd20 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.490+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.3 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fc918001910 con 0x7fc92809cd20 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.490+0000 7fc92f76b640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.100:6790/0 2024-09-17T15:54:18.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.490+0000 7fc92f76b640 1 -- 172.21.15.64:0/3319274594 --> v1:172.21.15.100:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fc9280a4fc0 con 0x7fc92809cd20 2024-09-17T15:54:18.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.490+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.3 v1:172.21.15.100:6790/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7fc91804ea70 con 0x7fc92809cd20 2024-09-17T15:54:18.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.490+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 --> v1:172.21.15.64:6800/3889666033 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc90c04b650 con 0x7fc90c0479e0 2024-09-17T15:54:18.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.490+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== mon.3 v1:172.21.15.100:6790/0 9 ==== mon_get_version_reply(handle=1 version=327) ==== 24+0+0 (unknown 900077626 0 0) 0x7fc91804ee80 con 0x7fc92809cd20 2024-09-17T15:54:18.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.490+0000 7fc91dffb640 10 monclient: handle_get_version_reply finishing 1 version 327 2024-09-17T15:54:18.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.490+0000 7fc92cd06640 10 monclient: get_auth_request con 0x7fc90c040de0 auth_method 0 2024-09-17T15:54:18.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.491+0000 7fc92dd08640 10 monclient: get_auth_request con 0x7fc90c0479e0 auth_method 0 2024-09-17T15:54:18.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.491+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== osd.1 v1:172.21.15.64:6800/3889666033 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc90c04b650 con 0x7fc90c0479e0 2024-09-17T15:54:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.504+0000 7fc92f76b640 1 -- 172.21.15.64:0/3319274594 --> v1:172.21.15.64:6800/3889666033 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc9280aa3f0 con 0x7fc90c0479e0 2024-09-17T15:54:18.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.505+0000 7fc91dffb640 1 -- 172.21.15.64:0/3319274594 <== osd.1 v1:172.21.15.64:6800/3889666033 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fc9280aa3f0 con 0x7fc90c0479e0 2024-09-17T15:54:18.515 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:18.515 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T15:54:18.515 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:18.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.505+0000 7fc92f76b640 1 -- 172.21.15.64:0/3319274594 >> v1:172.21.15.64:6800/3889666033 conn(0x7fc90c0479e0 legacy=0x7fc90c049e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.505+0000 7fc92f76b640 1 -- 172.21.15.64:0/3319274594 >> v1:172.21.15.100:6812/4210963484 conn(0x7fc90c040de0 legacy=0x7fc90c0432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.505+0000 7fc92f76b640 10 monclient: shutdown 2024-09-17T15:54:18.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.505+0000 7fc92f76b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:18.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.505+0000 7fc92f76b640 1 -- 172.21.15.64:0/3319274594 >> v1:172.21.15.100:6790/0 conn(0x7fc92809cd20 legacy=0x7fc9280a34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.505+0000 7fc92dd08640 1 -- 172.21.15.64:0/3319274594 reap_dead start 2024-09-17T15:54:18.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.505+0000 7fc92f76b640 1 -- 172.21.15.64:0/3319274594 shutdown_connections 2024-09-17T15:54:18.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.505+0000 7fc92f76b640 1 -- 172.21.15.64:0/3319274594 >> 172.21.15.64:0/3319274594 conn(0x7fc92806dc70 msgr2=0x7fc928134fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:18.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.506+0000 7fc92f76b640 1 -- 172.21.15.64:0/3319274594 shutdown_connections 2024-09-17T15:54:18.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.506+0000 7fc92f76b640 1 -- 172.21.15.64:0/3319274594 wait complete. 2024-09-17T15:54:18.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-17T15:54:18.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.619+0000 7f24099e2640 10 monclient: get_monmap_and_config 2024-09-17T15:54:18.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.619+0000 7f24099e2640 10 monclient: build_initial_monmap 2024-09-17T15:54:18.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.619+0000 7f24099e2640 10 monclient: monmap: 2024-09-17T15:54:18.622 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:18.622 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.622 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:18.620718+0000 2024-09-17T15:54:18.622 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:18.620718+0000 2024-09-17T15:54:18.622 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:18.622 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.622 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:18.622 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:18.622 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:18.622 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:18.622 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:18.622 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:18.623 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:18.623 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:18.623 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:18.623 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.620+0000 7f24099e2640 1 Processor -- start 2024-09-17T15:54:18.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.620+0000 7f24099e2640 1 -- start start 2024-09-17T15:54:18.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.620+0000 7f24099e2640 10 monclient: init 2024-09-17T15:54:18.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.620+0000 7f24099e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:18.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.620+0000 7f24099e2640 10 monclient: _add_conns ranks=[7,3,4,2,1,6,5,8,0] 2024-09-17T15:54:18.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.620+0000 7f24099e2640 10 monclient(hunting): picked mon.noname-i con 0x7f24040a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:18.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24099e2640 10 monclient(hunting): picked mon.noname-g con 0x7f2404135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:18.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24099e2640 10 monclient(hunting): picked mon.noname-c con 0x7f2404135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:18.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24099e2640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240409d6f0 con 0x7f2404135b50 2024-09-17T15:54:18.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24099e2640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240409d9d0 con 0x7f2404135f70 2024-09-17T15:54:18.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24099e2640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240409dcb0 con 0x7f24040a5520 2024-09-17T15:54:18.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24099e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:18.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24099e2640 10 monclient(hunting): authenticate will time out at 1256.296063s 2024-09-17T15:54:18.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24037fe640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7f2404135f70 0x7f24041415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:59860/0 (socket says 172.21.15.64:59860) 2024-09-17T15:54:18.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24037fe640 1 -- 172.21.15.64:0/471403556 learned_addr learned my addr 172.21.15.64:0/471403556 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:18.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 -- 172.21.15.64:0/471403556 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2153751258 0 0) 0x7f240409d9d0 con 0x7f2404135f70 2024-09-17T15:54:18.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient(hunting): my global_id is 154741 2024-09-17T15:54:18.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 -- 172.21.15.64:0/471403556 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23f0003520 con 0x7f2404135f70 2024-09-17T15:54:18.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 -- 172.21.15.64:0/471403556 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2659038003 0 0) 0x7f240409dcb0 con 0x7f24040a5520 2024-09-17T15:54:18.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient(hunting): my global_id is 174247 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 -- 172.21.15.64:0/471403556 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f240409d9d0 con 0x7f24040a5520 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 -- 172.21.15.64:0/471403556 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 374969459 0 0) 0x7f240409d6f0 con 0x7f2404135b50 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient(hunting): my global_id is 115995 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 -- 172.21.15.64:0/471403556 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f240409dcb0 con 0x7f2404135b50 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 -- 172.21.15.64:0/471403556 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3633453342 0 0) 0x7f23f0003520 con 0x7f2404135f70 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 -- 172.21.15.64:0/471403556 >> v1:172.21.15.100:6792/0 conn(0x7f24040a5520 legacy=0x7f24040a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 -- 172.21.15.64:0/471403556 >> v1:172.21.15.100:6790/0 conn(0x7f2404135b50 legacy=0x7f240413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 monclient: found mon.noname-c 2024-09-17T15:54:18.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:18.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:54:18.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 -- 172.21.15.64:0/471403556 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240409ccd0 con 0x7f2404135f70 2024-09-17T15:54:18.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient: _finish_auth 0 2024-09-17T15:54:18.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:54:18.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:18.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 -- 172.21.15.64:0/471403556 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f23f4001830 con 0x7f2404135f70 2024-09-17T15:54:18.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24099e2640 5 monclient: authenticate success, global_id 154741 2024-09-17T15:54:18.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24099e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:18.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:18.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient: dump: 2024-09-17T15:54:18.634 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.634 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.635 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24099e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:18.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 -- 172.21.15.64:0/471403556 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f23f40019d0 con 0x7f2404135f70 2024-09-17T15:54:18.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:18.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24099e2640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:18.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f24099e2640 4 set_mon_vals no callback set 2024-09-17T15:54:18.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 1 -- 172.21.15.64:0/471403556 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f23f4004be0 con 0x7f2404135f70 2024-09-17T15:54:18.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:18.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.621+0000 7f2401ffb640 10 monclient: dump: 2024-09-17T15:54:18.636 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.636 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.636 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.637 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.622+0000 7f24099e2640 10 monclient: shutdown 2024-09-17T15:54:18.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.622+0000 7f24099e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:18.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.622+0000 7f24099e2640 1 -- 172.21.15.64:0/471403556 >> v1:172.21.15.64:6791/0 conn(0x7f2404135f70 legacy=0x7f24041415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.622+0000 7f24099e2640 1 -- 172.21.15.64:0/471403556 shutdown_connections 2024-09-17T15:54:18.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.622+0000 7f24099e2640 1 -- 172.21.15.64:0/471403556 >> 172.21.15.64:0/471403556 conn(0x7f240406db30 msgr2=0x7f24040ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:18.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.622+0000 7f24099e2640 1 -- 172.21.15.64:0/471403556 shutdown_connections 2024-09-17T15:54:18.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.622+0000 7f24099e2640 1 -- 172.21.15.64:0/471403556 wait complete. 2024-09-17T15:54:18.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.622+0000 7f24099e2640 10 monclient: build_initial_monmap 2024-09-17T15:54:18.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.622+0000 7f24099e2640 10 monclient: monmap: 2024-09-17T15:54:18.638 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:18.638 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:18.638 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:18.638 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f24099e2640 1 Processor -- start 2024-09-17T15:54:18.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f24099e2640 1 -- start start 2024-09-17T15:54:18.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f24099e2640 10 monclient: init 2024-09-17T15:54:18.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f24099e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:18.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f24099e2640 10 monclient: _add_conns ranks=[8,1,7,4,2,6,0,5,3] 2024-09-17T15:54:18.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f24099e2640 10 monclient(hunting): picked mon.noname-e con 0x7f24040a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:18.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f24099e2640 10 monclient(hunting): picked mon.noname-f con 0x7f2404135b50 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:18.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f24099e2640 10 monclient(hunting): picked mon.noname-i con 0x7f2404135f70 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:18.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f24099e2640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2404138b20 con 0x7f2404135b50 2024-09-17T15:54:18.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f24099e2640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240413a5e0 con 0x7f2404135f70 2024-09-17T15:54:18.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f24099e2640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240413a8c0 con 0x7f24040a5520 2024-09-17T15:54:18.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f24099e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:18.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f24099e2640 10 monclient(hunting): authenticate will time out at 1256.298934s 2024-09-17T15:54:18.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f2402ffd640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f24040a5520 0x7f24041370c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51112/0 (socket says 172.21.15.64:51112) 2024-09-17T15:54:18.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.623+0000 7f2402ffd640 1 -- 172.21.15.64:0/1229555992 learned_addr learned my addr 172.21.15.64:0/1229555992 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:18.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 40115668 0 0) 0x7f240413a8c0 con 0x7f24040a5520 2024-09-17T15:54:18.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): my global_id is 174230 2024-09-17T15:54:18.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23e4003550 con 0x7f24040a5520 2024-09-17T15:54:18.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2838121986 0 0) 0x7f240413a5e0 con 0x7f2404135f70 2024-09-17T15:54:18.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): my global_id is 174256 2024-09-17T15:54:18.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f240413a8c0 con 0x7f2404135f70 2024-09-17T15:54:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 474869530 0 0) 0x7f2404138b20 con 0x7f2404135b50 2024-09-17T15:54:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): my global_id is 6994 2024-09-17T15:54:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f240413a5e0 con 0x7f2404135b50 2024-09-17T15:54:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1633220308 0 0) 0x7f23e4003550 con 0x7f24040a5520 2024-09-17T15:54:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2404138b20 con 0x7f24040a5520 2024-09-17T15:54:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f23fc001f50 con 0x7f24040a5520 2024-09-17T15:54:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:54:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): dump: 2024-09-17T15:54:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.643 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.643 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.643 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.643 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.643 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.643 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.643 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.643 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.643 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.643 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.643 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.643 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.644 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.644 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.644 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:18.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2993754499 0 0) 0x7f240413a8c0 con 0x7f2404135f70 2024-09-17T15:54:18.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23e4003550 con 0x7f2404135f70 2024-09-17T15:54:18.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f23f4001770 con 0x7f2404135f70 2024-09-17T15:54:18.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:18.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:54:18.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): dump: 2024-09-17T15:54:18.644 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.646 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.646 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 171117525 0 0) 0x7f240413a5e0 con 0x7f2404135b50 2024-09-17T15:54:18.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f240413a8c0 con 0x7f2404135b50 2024-09-17T15:54:18.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f23ec0019d0 con 0x7f2404135b50 2024-09-17T15:54:18.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:18.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:54:18.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient(hunting): dump: 2024-09-17T15:54:18.646 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.646 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.646 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.646 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.647 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.647 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.647 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.647 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.647 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.647 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.647 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.647 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.647 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.647 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.647 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.647 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2179123567 0 0) 0x7f2404138b20 con 0x7f24040a5520 2024-09-17T15:54:18.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 >> v1:172.21.15.100:6792/0 conn(0x7f2404135f70 legacy=0x7f24041d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 >> v1:172.21.15.100:6789/0 conn(0x7f2404135b50 legacy=0x7f24041377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient: _finish_hunting 0 2024-09-17T15:54:18.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 monclient: found mon.e 2024-09-17T15:54:18.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:18.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:18.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24040a67e0 con 0x7f24040a5520 2024-09-17T15:54:18.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient: _finish_auth 0 2024-09-17T15:54:18.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:54:18.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:18.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f24099e2640 5 monclient: authenticate success, global_id 174230 2024-09-17T15:54:18.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f24099e2640 10 monclient: _renew_subs 2024-09-17T15:54:18.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f24099e2640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:18.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f24099e2640 1 -- 172.21.15.64:0/1229555992 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2404137ee0 con 0x7f24040a5520 2024-09-17T15:54:18.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f24099e2640 10 monclient: _renew_subs 2024-09-17T15:54:18.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f24099e2640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:18.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f24099e2640 1 -- 172.21.15.64:0/1229555992 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f24041424d0 con 0x7f24040a5520 2024-09-17T15:54:18.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f23fc003c40 con 0x7f24040a5520 2024-09-17T15:54:18.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:18.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f23fc0041f0 con 0x7f24040a5520 2024-09-17T15:54:18.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f24017fa640 4 set_mon_vals no callback set 2024-09-17T15:54:18.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:18.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.624+0000 7f23fbfff640 10 monclient: dump: 2024-09-17T15:54:18.649 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.651 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.625+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f23fc004410 con 0x7f24040a5520 2024-09-17T15:54:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.625+0000 7f24099e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.625+0000 7f24099e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.625+0000 7f24099e2640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.625+0000 7f24099e2640 1 -- 172.21.15.64:0/1229555992 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23e8005080 con 0x7f24040a5520 2024-09-17T15:54:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.626+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f23fc04edb0 con 0x7f24040a5520 2024-09-17T15:54:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.626+0000 7f24027fc640 10 monclient: get_auth_request con 0x7f23e4040d30 auth_method 0 2024-09-17T15:54:18.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.630+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mon.8 v1:172.21.15.64:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f23fc017c70 con 0x7f24040a5520 2024-09-17T15:54:18.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.630+0000 7f23fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:18.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.630+0000 7f23fbfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:18.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.813+0000 7f24099e2640 1 -- 172.21.15.64:0/1229555992 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f23e8000b90 con 0x7f23e4040d30 2024-09-17T15:54:18.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.814+0000 7f23fbfff640 1 -- 172.21.15.64:0/1229555992 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (unknown 489605429 0 0) 0x7f23e8000b90 con 0x7f23e4040d30 2024-09-17T15:54:18.816 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 1 to deep-scrub 2024-09-17T15:54:18.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.816+0000 7f24099e2640 1 -- 172.21.15.64:0/1229555992 >> v1:172.21.15.100:6812/4210963484 conn(0x7f23e4040d30 legacy=0x7f23e40431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.816+0000 7f24099e2640 10 monclient: shutdown 2024-09-17T15:54:18.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.816+0000 7f24099e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:18.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.816+0000 7f24099e2640 1 -- 172.21.15.64:0/1229555992 >> v1:172.21.15.64:6793/0 conn(0x7f24040a5520 legacy=0x7f24041370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.816+0000 7f24099e2640 1 -- 172.21.15.64:0/1229555992 shutdown_connections 2024-09-17T15:54:18.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.816+0000 7f24099e2640 1 -- 172.21.15.64:0/1229555992 >> 172.21.15.64:0/1229555992 conn(0x7f240406db30 msgr2=0x7f24040a77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:18.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.816+0000 7f24099e2640 1 -- 172.21.15.64:0/1229555992 shutdown_connections 2024-09-17T15:54:18.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.816+0000 7f24099e2640 1 -- 172.21.15.64:0/1229555992 wait complete. 2024-09-17T15:54:18.828 INFO:tasks.ceph:Scrubbing osd.2 2024-09-17T15:54:18.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-17T15:54:18.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.932+0000 7f09cd42a640 10 monclient: get_monmap_and_config 2024-09-17T15:54:18.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.932+0000 7f09cd42a640 10 monclient: build_initial_monmap 2024-09-17T15:54:18.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.932+0000 7f09cd42a640 10 monclient: monmap: 2024-09-17T15:54:18.934 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:18.934 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.934 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:18.933191+0000 2024-09-17T15:54:18.934 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:18.933191+0000 2024-09-17T15:54:18.934 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:18.934 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.935 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:18.935 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:18.935 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:18.935 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:18.935 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:18.935 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:18.935 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:18.935 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:18.935 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:18.935 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.932+0000 7f09cd42a640 1 Processor -- start 2024-09-17T15:54:18.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09cd42a640 1 -- start start 2024-09-17T15:54:18.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09cd42a640 10 monclient: init 2024-09-17T15:54:18.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09cd42a640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:18.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09cd42a640 10 monclient: _add_conns ranks=[7,1,3,2,5,4,6,8,0] 2024-09-17T15:54:18.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09cd42a640 10 monclient(hunting): picked mon.noname-i con 0x7f09c8138d80 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:18.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09cd42a640 10 monclient(hunting): picked mon.noname-f con 0x7f09c80a5de0 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:18.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09cd42a640 10 monclient(hunting): picked mon.noname-g con 0x7f09c809cd20 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:18.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09cd42a640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c80a8390 con 0x7f09c80a5de0 2024-09-17T15:54:18.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09cd42a640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c809db80 con 0x7f09c809cd20 2024-09-17T15:54:18.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09cd42a640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c809de60 con 0x7f09c8138d80 2024-09-17T15:54:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09cd42a640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09cd42a640 10 monclient(hunting): authenticate will time out at 1256.608656s 2024-09-17T15:54:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09c77fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f09c809cd20 0x7f09c8141830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:57852/0 (socket says 172.21.15.64:57852) 2024-09-17T15:54:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09c67fc640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f09c80a5de0 0x7f09c813e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.64:36406/0 (socket says 172.21.15.64:36406) 2024-09-17T15:54:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.933+0000 7f09c77fe640 1 -- 172.21.15.64:0/2559063214 learned_addr learned my addr 172.21.15.64:0/2559063214 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 -- 172.21.15.64:0/2559063214 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3534814748 0 0) 0x7f09c80a8390 con 0x7f09c80a5de0 2024-09-17T15:54:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient(hunting): my global_id is 7003 2024-09-17T15:54:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 -- 172.21.15.64:0/2559063214 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09b0003520 con 0x7f09c80a5de0 2024-09-17T15:54:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 -- 172.21.15.64:0/2559063214 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3098841597 0 0) 0x7f09c809de60 con 0x7f09c8138d80 2024-09-17T15:54:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient(hunting): my global_id is 174265 2024-09-17T15:54:18.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 -- 172.21.15.64:0/2559063214 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09c80a8390 con 0x7f09c8138d80 2024-09-17T15:54:18.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 -- 172.21.15.64:0/2559063214 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3482550822 0 0) 0x7f09c809db80 con 0x7f09c809cd20 2024-09-17T15:54:18.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient(hunting): my global_id is 116004 2024-09-17T15:54:18.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 -- 172.21.15.64:0/2559063214 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09c809de60 con 0x7f09c809cd20 2024-09-17T15:54:18.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 -- 172.21.15.64:0/2559063214 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 445327435 0 0) 0x7f09b0003520 con 0x7f09c80a5de0 2024-09-17T15:54:18.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 -- 172.21.15.64:0/2559063214 >> v1:172.21.15.100:6792/0 conn(0x7f09c8138d80 legacy=0x7f09c80a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 -- 172.21.15.64:0/2559063214 >> v1:172.21.15.100:6790/0 conn(0x7f09c809cd20 legacy=0x7f09c8141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:54:18.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 monclient: found mon.noname-f 2024-09-17T15:54:18.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:18.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.100:6789/0 2024-09-17T15:54:18.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 -- 172.21.15.64:0/2559063214 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c809e140 con 0x7f09c80a5de0 2024-09-17T15:54:18.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:54:18.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:54:18.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:18.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 -- 172.21.15.64:0/2559063214 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f09bc001830 con 0x7f09c80a5de0 2024-09-17T15:54:18.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-17T15:54:18.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient: dump: 2024-09-17T15:54:18.947 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.947 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.947 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.947 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.947 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.947 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.947 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.947 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09cd42a640 5 monclient: authenticate success, global_id 7003 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09cd42a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 -- 172.21.15.64:0/2559063214 <== mon.1 v1:172.21.15.100:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f09bc0019d0 con 0x7f09c80a5de0 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 1 -- 172.21.15.64:0/2559063214 <== mon.1 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f09bc004b20 con 0x7f09c80a5de0 2024-09-17T15:54:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09c5ffb640 10 monclient: dump: 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.949 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.950 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09cd42a640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.934+0000 7f09cd42a640 4 set_mon_vals no callback set 2024-09-17T15:54:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.935+0000 7f09cd42a640 10 monclient: shutdown 2024-09-17T15:54:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.935+0000 7f09cd42a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.935+0000 7f09cd42a640 1 -- 172.21.15.64:0/2559063214 >> v1:172.21.15.100:6789/0 conn(0x7f09c80a5de0 legacy=0x7f09c813e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.935+0000 7f09cd42a640 1 -- 172.21.15.64:0/2559063214 shutdown_connections 2024-09-17T15:54:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.935+0000 7f09cd42a640 1 -- 172.21.15.64:0/2559063214 >> 172.21.15.64:0/2559063214 conn(0x7f09c806dc70 msgr2=0x7f09c80abb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.935+0000 7f09cd42a640 1 -- 172.21.15.64:0/2559063214 shutdown_connections 2024-09-17T15:54:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.935+0000 7f09cd42a640 1 -- 172.21.15.64:0/2559063214 wait complete. 2024-09-17T15:54:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.935+0000 7f09cd42a640 10 monclient: build_initial_monmap 2024-09-17T15:54:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.935+0000 7f09cd42a640 10 monclient: monmap: 2024-09-17T15:54:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:18.952 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:18.952 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:18.952 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:18.952 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:18.952 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:18.952 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:18.952 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:18.952 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09cd42a640 1 Processor -- start 2024-09-17T15:54:18.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09cd42a640 1 -- start start 2024-09-17T15:54:18.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09cd42a640 10 monclient: init 2024-09-17T15:54:18.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09cd42a640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:18.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09cd42a640 10 monclient: _add_conns ranks=[0,3,5,4,8,1,6,2,7] 2024-09-17T15:54:18.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09cd42a640 10 monclient(hunting): picked mon.noname-a con 0x7f09c809cd20 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:18.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09cd42a640 10 monclient(hunting): picked mon.noname-g con 0x7f09c80a5de0 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:18.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09cd42a640 10 monclient(hunting): picked mon.noname-h con 0x7f09c8138d80 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:18.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09cd42a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c812ff90 con 0x7f09c809cd20 2024-09-17T15:54:18.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09cd42a640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c8130580 con 0x7f09c80a5de0 2024-09-17T15:54:18.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09cd42a640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c80a9cb0 con 0x7f09c8138d80 2024-09-17T15:54:18.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09cd42a640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:18.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09cd42a640 10 monclient(hunting): authenticate will time out at 1256.611830s 2024-09-17T15:54:18.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09c6ffd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f09c809cd20 0x7f09c812ea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35334/0 (socket says 172.21.15.64:35334) 2024-09-17T15:54:18.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.936+0000 7f09c6ffd640 1 -- 172.21.15.64:0/1714326173 learned_addr learned my addr 172.21.15.64:0/1714326173 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3567020380 0 0) 0x7f09c812ff90 con 0x7f09c809cd20 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): my global_id is 125190 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09b4003520 con 0x7f09c809cd20 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3424024480 0 0) 0x7f09b4003520 con 0x7f09c809cd20 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09c812ff90 con 0x7f09c809cd20 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2161037962 0 0) 0x7f09c80a9cb0 con 0x7f09c8138d80 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): my global_id is 174308 2024-09-17T15:54:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09b4003520 con 0x7f09c8138d80 2024-09-17T15:54:18.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f09b8001b50 con 0x7f09c809cd20 2024-09-17T15:54:18.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:18.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:54:18.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): dump: 2024-09-17T15:54:18.955 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.955 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.955 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.955 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.955 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.955 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.955 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.955 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.955 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3332460265 0 0) 0x7f09c8130580 con 0x7f09c80a5de0 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient(hunting): my global_id is 116013 2024-09-17T15:54:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09c80a9cb0 con 0x7f09c80a5de0 2024-09-17T15:54:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2708990966 0 0) 0x7f09c812ff90 con 0x7f09c809cd20 2024-09-17T15:54:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 >> v1:172.21.15.100:6791/0 conn(0x7f09c8138d80 legacy=0x7f09c81d6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 >> v1:172.21.15.100:6790/0 conn(0x7f09c80a5de0 legacy=0x7f09c812f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:54:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 monclient: found mon.a 2024-09-17T15:54:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c812f8b0 con 0x7f09c809cd20 2024-09-17T15:54:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient: _finish_auth 0 2024-09-17T15:54:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:54:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09cd42a640 5 monclient: authenticate success, global_id 125190 2024-09-17T15:54:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09cd42a640 10 monclient: _renew_subs 2024-09-17T15:54:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09cd42a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09cd42a640 1 -- 172.21.15.64:0/1714326173 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f09c81328e0 con 0x7f09c809cd20 2024-09-17T15:54:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09cd42a640 10 monclient: _renew_subs 2024-09-17T15:54:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09cd42a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09cd42a640 1 -- 172.21.15.64:0/1714326173 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f09c8131350 con 0x7f09c809cd20 2024-09-17T15:54:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f09b8001cf0 con 0x7f09c809cd20 2024-09-17T15:54:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f09b8003c60 con 0x7f09c809cd20 2024-09-17T15:54:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09c57fa640 4 set_mon_vals no callback set 2024-09-17T15:54:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.937+0000 7f09af7fe640 10 monclient: dump: 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:18.959 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:18.960 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:18.960 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:18.960 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:18.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.938+0000 7f09cd42a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:18.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.938+0000 7f09cd42a640 1 -- 172.21.15.64:0/1714326173 --> v1:172.21.15.64:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f09a8000f80 con 0x7f09c809cd20 2024-09-17T15:54:18.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.938+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f09b8003e80 con 0x7f09c809cd20 2024-09-17T15:54:18.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.938+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f09b804ead0 con 0x7f09c809cd20 2024-09-17T15:54:18.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.939+0000 7f09c67fc640 10 monclient: get_auth_request con 0x7f09b4040540 auth_method 0 2024-09-17T15:54:18.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.939+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 --> v1:172.21.15.64:6804/1825798494 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09b404ad40 con 0x7f09b40470d0 2024-09-17T15:54:18.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.939+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_get_version_reply(handle=1 version=327) ==== 24+0+0 (unknown 900077626 0 0) 0x7f09b804ef40 con 0x7f09c809cd20 2024-09-17T15:54:18.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.939+0000 7f09af7fe640 10 monclient: handle_get_version_reply finishing 1 version 327 2024-09-17T15:54:18.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.939+0000 7f09c77fe640 10 monclient: get_auth_request con 0x7f09b40470d0 auth_method 0 2024-09-17T15:54:18.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.939+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 <== osd.2 v1:172.21.15.64:6804/1825798494 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f09b404ad40 con 0x7f09b40470d0 2024-09-17T15:54:18.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.952+0000 7f09cd42a640 1 -- 172.21.15.64:0/1714326173 --> v1:172.21.15.64:6804/1825798494 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f09a8002cd0 con 0x7f09b40470d0 2024-09-17T15:54:18.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.952+0000 7f09af7fe640 1 -- 172.21.15.64:0/1714326173 <== osd.2 v1:172.21.15.64:6804/1825798494 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f09a8002cd0 con 0x7f09b40470d0 2024-09-17T15:54:18.962 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:18.962 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T15:54:18.962 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:18.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.952+0000 7f09cd42a640 1 -- 172.21.15.64:0/1714326173 >> v1:172.21.15.64:6804/1825798494 conn(0x7f09b40470d0 legacy=0x7f09b40494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.953+0000 7f09cd42a640 1 -- 172.21.15.64:0/1714326173 >> v1:172.21.15.100:6812/4210963484 conn(0x7f09b4040540 legacy=0x7f09b4042a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.953+0000 7f09cd42a640 10 monclient: shutdown 2024-09-17T15:54:18.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.953+0000 7f09cd42a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:18.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.953+0000 7f09cd42a640 1 -- 172.21.15.64:0/1714326173 >> v1:172.21.15.64:6789/0 conn(0x7f09c809cd20 legacy=0x7f09c812ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:18.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.953+0000 7f09c77fe640 1 -- 172.21.15.64:0/1714326173 reap_dead start 2024-09-17T15:54:18.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.953+0000 7f09cd42a640 1 -- 172.21.15.64:0/1714326173 shutdown_connections 2024-09-17T15:54:18.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.953+0000 7f09cd42a640 1 -- 172.21.15.64:0/1714326173 >> 172.21.15.64:0/1714326173 conn(0x7f09c806dc70 msgr2=0x7f09c80a6400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:18.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.953+0000 7f09cd42a640 1 -- 172.21.15.64:0/1714326173 shutdown_connections 2024-09-17T15:54:18.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:18.953+0000 7f09cd42a640 1 -- 172.21.15.64:0/1714326173 wait complete. 2024-09-17T15:54:18.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-17T15:54:19.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.064+0000 7f501c5b4640 10 monclient: get_monmap_and_config 2024-09-17T15:54:19.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.064+0000 7f501c5b4640 10 monclient: build_initial_monmap 2024-09-17T15:54:19.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.064+0000 7f501c5b4640 10 monclient: monmap: 2024-09-17T15:54:19.067 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:19.067 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.067 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:19.065696+0000 2024-09-17T15:54:19.067 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:19.065696+0000 2024-09-17T15:54:19.067 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:19.067 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.067 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:19.067 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:19.067 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:19.068 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:19.068 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:19.068 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:19.068 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:19.068 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:19.068 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:19.068 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.065+0000 7f501c5b4640 1 Processor -- start 2024-09-17T15:54:19.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.065+0000 7f501c5b4640 1 -- start start 2024-09-17T15:54:19.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.065+0000 7f501c5b4640 10 monclient: init 2024-09-17T15:54:19.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:19.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 10 monclient: _add_conns ranks=[1,7,0,6,2,8,4,3,5] 2024-09-17T15:54:19.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 10 monclient(hunting): picked mon.noname-f con 0x7f50140a5520 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:19.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 10 monclient(hunting): picked mon.noname-i con 0x7f5014135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:19.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 10 monclient(hunting): picked mon.noname-a con 0x7f5014135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:19.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501409d6f0 con 0x7f5014135f70 2024-09-17T15:54:19.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501409d9d0 con 0x7f50140a5520 2024-09-17T15:54:19.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501409dcb0 con 0x7f5014135b50 2024-09-17T15:54:19.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:19.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 10 monclient(hunting): authenticate will time out at 1256.741241s 2024-09-17T15:54:19.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501ab51640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f5014135f70 0x7f50141415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35338/0 (socket says 172.21.15.64:35338) 2024-09-17T15:54:19.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501ab51640 1 -- 172.21.15.64:0/2067456026 learned_addr learned my addr 172.21.15.64:0/2067456026 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:19.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 1 -- 172.21.15.64:0/2067456026 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1005679438 0 0) 0x7f501409d6f0 con 0x7f5014135f70 2024-09-17T15:54:19.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient(hunting): my global_id is 125199 2024-09-17T15:54:19.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 1 -- 172.21.15.64:0/2067456026 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5000003520 con 0x7f5014135f70 2024-09-17T15:54:19.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501a350640 1 -- 172.21.15.64:0/2067456026 >> v1:172.21.15.100:6789/0 conn(0x7f50140a5520 legacy=0x7f50140a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-17T15:54:19.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501a350640 1 -- 172.21.15.64:0/2067456026 >> v1:172.21.15.100:6789/0 conn(0x7f50140a5520 legacy=0x7f50140a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:54:19.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501a350640 1 --1- 172.21.15.64:0/2067456026 >> v1:172.21.15.100:6789/0 conn(0x7f50140a5520 0x7f50140a5900 :-1 s=OPENED pgs=724 cs=1 l=1).handle_message read tag failed 2024-09-17T15:54:19.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501a350640 1 --1- 172.21.15.64:0/2067456026 >> v1:172.21.15.100:6789/0 conn(0x7f50140a5520 0x7f50140a5900 :-1 s=OPENED pgs=724 cs=1 l=1).fault on lossy channel, failing 2024-09-17T15:54:19.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.100:6789/0 2024-09-17T15:54:19.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 1 -- 172.21.15.64:0/2067456026 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3218473147 0 0) 0x7f5000003520 con 0x7f5014135f70 2024-09-17T15:54:19.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 1 -- 172.21.15.64:0/2067456026 >> v1:172.21.15.100:6792/0 conn(0x7f5014135b50 legacy=0x7f501413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 1 -- 172.21.15.64:0/2067456026 >> v1:172.21.15.100:6789/0 conn(0x7f50140a5520 legacy=0x7f50140a5900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:54:19.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient: _finish_hunting 0 2024-09-17T15:54:19.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 1 monclient: found mon.noname-a 2024-09-17T15:54:19.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:19.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:54:19.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 1 -- 172.21.15.64:0/2067456026 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501409ccd0 con 0x7f5014135f70 2024-09-17T15:54:19.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient: _finish_auth 0 2024-09-17T15:54:19.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient: _check_auth_tickets 2024-09-17T15:54:19.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:19.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 1 -- 172.21.15.64:0/2067456026 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f500c001bb0 con 0x7f5014135f70 2024-09-17T15:54:19.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 5 monclient: authenticate success, global_id 125199 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient: dump: 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.082 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.083 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.083 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.083 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.083 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.083 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.083 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:19.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 1 -- 172.21.15.64:0/2067456026 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f500c001d50 con 0x7f5014135f70 2024-09-17T15:54:19.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:19.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 1 -- 172.21.15.64:0/2067456026 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f500c004f60 con 0x7f5014135f70 2024-09-17T15:54:19.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:19.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501c5b4640 4 set_mon_vals no callback set 2024-09-17T15:54:19.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:19.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.066+0000 7f501934e640 10 monclient: dump: 2024-09-17T15:54:19.084 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.084 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.084 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.084 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.084 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.084 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.067+0000 7f501c5b4640 10 monclient: shutdown 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.067+0000 7f501c5b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.067+0000 7f501c5b4640 1 -- 172.21.15.64:0/2067456026 >> v1:172.21.15.64:6789/0 conn(0x7f5014135f70 legacy=0x7f50141415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.067+0000 7f501c5b4640 1 -- 172.21.15.64:0/2067456026 shutdown_connections 2024-09-17T15:54:19.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.067+0000 7f501c5b4640 1 -- 172.21.15.64:0/2067456026 >> 172.21.15.64:0/2067456026 conn(0x7f501406db30 msgr2=0x7f50140ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:19.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.067+0000 7f501c5b4640 1 -- 172.21.15.64:0/2067456026 shutdown_connections 2024-09-17T15:54:19.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.067+0000 7f501c5b4640 1 -- 172.21.15.64:0/2067456026 wait complete. 2024-09-17T15:54:19.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.068+0000 7f501c5b4640 10 monclient: build_initial_monmap 2024-09-17T15:54:19.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.068+0000 7f501c5b4640 10 monclient: monmap: 2024-09-17T15:54:19.086 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:19.086 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:19.086 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:19.086 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:19.086 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:19.086 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:19.086 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:19.086 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:19.087 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.068+0000 7f501c5b4640 1 Processor -- start 2024-09-17T15:54:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.068+0000 7f501c5b4640 1 -- start start 2024-09-17T15:54:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.068+0000 7f501c5b4640 10 monclient: init 2024-09-17T15:54:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.068+0000 7f501c5b4640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.068+0000 7f501c5b4640 10 monclient: _add_conns ranks=[8,7,6,1,5,3,2,0,4] 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.068+0000 7f501c5b4640 10 monclient(hunting): picked mon.noname-e con 0x7f50140a5520 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f501c5b4640 10 monclient(hunting): picked mon.noname-i con 0x7f5014135b50 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f501c5b4640 10 monclient(hunting): picked mon.noname-d con 0x7f5014135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f501c5b4640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5014141ac0 con 0x7f5014135f70 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f501c5b4640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501413ef50 con 0x7f5014135b50 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f501c5b4640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5014141da0 con 0x7f50140a5520 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f501c5b4640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f501c5b4640 10 monclient(hunting): authenticate will time out at 1256.744094s 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f501a350640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f50140a5520 0x7f501413e130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51116/0 (socket says 172.21.15.64:51116) 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f501a350640 1 -- 172.21.15.64:0/2639165 learned_addr learned my addr 172.21.15.64:0/2639165 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2328735392 0 0) 0x7f5014141da0 con 0x7f50140a5520 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): my global_id is 174239 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ff8003550 con 0x7f50140a5520 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3332036629 0 0) 0x7f5014141ac0 con 0x7f5014135f70 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): my global_id is 174255 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5014141da0 con 0x7f5014135f70 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1609702344 0 0) 0x7f501413ef50 con 0x7f5014135b50 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): my global_id is 174283 2024-09-17T15:54:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5014141ac0 con 0x7f5014135b50 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3321968668 0 0) 0x7f4ff8003550 con 0x7f50140a5520 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f501413ef50 con 0x7f50140a5520 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5010001c70 con 0x7f50140a5520 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): dump: 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2392471245 0 0) 0x7f5014141da0 con 0x7f5014135f70 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ff8003550 con 0x7f5014135f70 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f500c001af0 con 0x7f5014135f70 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): dump: 2024-09-17T15:54:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.093 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.093 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.094 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.094 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.094 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3436686921 0 0) 0x7f5014141ac0 con 0x7f5014135b50 2024-09-17T15:54:19.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5014141da0 con 0x7f5014135b50 2024-09-17T15:54:19.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5004001980 con 0x7f5014135b50 2024-09-17T15:54:19.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:19.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:54:19.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 10 monclient(hunting): dump: 2024-09-17T15:54:19.094 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.095 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.096 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.069+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2256525123 0 0) 0x7f501413ef50 con 0x7f50140a5520 2024-09-17T15:54:19.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 >> v1:172.21.15.100:6792/0 conn(0x7f5014135b50 legacy=0x7f501413e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 >> v1:172.21.15.64:6792/0 conn(0x7f5014135f70 legacy=0x7f50141d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 10 monclient: _finish_hunting 0 2024-09-17T15:54:19.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 1 monclient: found mon.e 2024-09-17T15:54:19.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:19.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:19.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50140a6490 con 0x7f50140a5520 2024-09-17T15:54:19.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 10 monclient: _finish_auth 0 2024-09-17T15:54:19.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 10 monclient: _check_auth_tickets 2024-09-17T15:54:19.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:19.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f501c5b4640 5 monclient: authenticate success, global_id 174239 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f501c5b4640 10 monclient: _renew_subs 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f501c5b4640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f501c5b4640 1 -- 172.21.15.64:0/2639165 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f50140a9040 con 0x7f50140a5520 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f501c5b4640 10 monclient: _renew_subs 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f501c5b4640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f501c5b4640 1 -- 172.21.15.64:0/2639165 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f5014142560 con 0x7f50140a5520 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f50100038a0 con 0x7f50140a5520 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f5010003e10 con 0x7f50140a5520 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f5018b4d640 4 set_mon_vals no callback set 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.070+0000 7f500affd640 10 monclient: dump: 2024-09-17T15:54:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.099 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.071+0000 7f501c5b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.071+0000 7f501c5b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.071+0000 7f501c5b4640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.071+0000 7f501c5b4640 1 -- 172.21.15.64:0/2639165 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ffc005080 con 0x7f50140a5520 2024-09-17T15:54:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.071+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f5010003fb0 con 0x7f50140a5520 2024-09-17T15:54:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.071+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f501004eb20 con 0x7f50140a5520 2024-09-17T15:54:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.072+0000 7f5019b4f640 10 monclient: get_auth_request con 0x7f4ff8040da0 auth_method 0 2024-09-17T15:54:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.076+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mon.8 v1:172.21.15.64:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f5010017910 con 0x7f50140a5520 2024-09-17T15:54:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.076+0000 7f500affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.076+0000 7f500affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:19.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.259+0000 7f501c5b4640 1 -- 172.21.15.64:0/2639165 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f4ffc000b90 con 0x7f4ff8040da0 2024-09-17T15:54:19.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.259+0000 7f500affd640 1 -- 172.21.15.64:0/2639165 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (unknown 3049115190 0 0) 0x7f4ffc000b90 con 0x7f4ff8040da0 2024-09-17T15:54:19.261 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 2 to deep-scrub 2024-09-17T15:54:19.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.261+0000 7f501c5b4640 1 -- 172.21.15.64:0/2639165 >> v1:172.21.15.100:6812/4210963484 conn(0x7f4ff8040da0 legacy=0x7f4ff8043260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.261+0000 7f501c5b4640 10 monclient: shutdown 2024-09-17T15:54:19.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.261+0000 7f501c5b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:19.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.261+0000 7f501c5b4640 1 -- 172.21.15.64:0/2639165 >> v1:172.21.15.64:6793/0 conn(0x7f50140a5520 legacy=0x7f501413e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.261+0000 7f501c5b4640 1 -- 172.21.15.64:0/2639165 shutdown_connections 2024-09-17T15:54:19.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.261+0000 7f501c5b4640 1 -- 172.21.15.64:0/2639165 >> 172.21.15.64:0/2639165 conn(0x7f501406db30 msgr2=0x7f50141334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:19.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.261+0000 7f501c5b4640 1 -- 172.21.15.64:0/2639165 shutdown_connections 2024-09-17T15:54:19.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.262+0000 7f501c5b4640 1 -- 172.21.15.64:0/2639165 wait complete. 2024-09-17T15:54:19.274 INFO:tasks.ceph:Scrubbing osd.3 2024-09-17T15:54:19.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-17T15:54:19.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.377+0000 7fb67af0e640 10 monclient: get_monmap_and_config 2024-09-17T15:54:19.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.377+0000 7fb67af0e640 10 monclient: build_initial_monmap 2024-09-17T15:54:19.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.377+0000 7fb67af0e640 10 monclient: monmap: 2024-09-17T15:54:19.379 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:19.379 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.379 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:19.378352+0000 2024-09-17T15:54:19.379 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:19.378352+0000 2024-09-17T15:54:19.379 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:19.379 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.380 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:19.380 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:19.380 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:19.380 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:19.380 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:19.380 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:19.380 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:19.380 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:19.380 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:19.380 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.378+0000 7fb67af0e640 1 Processor -- start 2024-09-17T15:54:19.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.378+0000 7fb67af0e640 1 -- start start 2024-09-17T15:54:19.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.378+0000 7fb67af0e640 10 monclient: init 2024-09-17T15:54:19.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.378+0000 7fb67af0e640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:19.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.378+0000 7fb67af0e640 10 monclient: _add_conns ranks=[3,5,7,2,6,8,0,4,1] 2024-09-17T15:54:19.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.378+0000 7fb67af0e640 10 monclient(hunting): picked mon.noname-g con 0x7fb674138d80 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:19.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.378+0000 7fb67af0e640 10 monclient(hunting): picked mon.noname-h con 0x7fb6740a5de0 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:19.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.378+0000 7fb67af0e640 10 monclient(hunting): picked mon.noname-i con 0x7fb67409cd20 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.378+0000 7fb67af0e640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6740a8390 con 0x7fb674138d80 2024-09-17T15:54:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.378+0000 7fb67af0e640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb67409db80 con 0x7fb6740a5de0 2024-09-17T15:54:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.378+0000 7fb67af0e640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb67409de60 con 0x7fb67409cd20 2024-09-17T15:54:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.378+0000 7fb67af0e640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.378+0000 7fb67af0e640 10 monclient(hunting): authenticate will time out at 1257.053827s 2024-09-17T15:54:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb678caa640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fb674138d80 0x7fb6740a56d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.64:57870/0 (socket says 172.21.15.64:57870) 2024-09-17T15:54:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb678caa640 1 -- 172.21.15.64:0/1364235811 learned_addr learned my addr 172.21.15.64:0/1364235811 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb673fff640 1 --1- 172.21.15.64:0/1364235811 >> v1:172.21.15.100:6791/0 conn(0x7fb6740a5de0 0x7fb67413e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6791/0 says I am v1:172.21.15.64:50916/0 (socket says 172.21.15.64:50916) 2024-09-17T15:54:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 -- 172.21.15.64:0/1364235811 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3906797049 0 0) 0x7fb67409de60 con 0x7fb67409cd20 2024-09-17T15:54:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient(hunting): my global_id is 174292 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 -- 172.21.15.64:0/1364235811 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb664003520 con 0x7fb67409cd20 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 -- 172.21.15.64:0/1364235811 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1254780518 0 0) 0x7fb67409db80 con 0x7fb6740a5de0 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient(hunting): my global_id is 174317 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 -- 172.21.15.64:0/1364235811 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb67409de60 con 0x7fb6740a5de0 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 -- 172.21.15.64:0/1364235811 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3758218280 0 0) 0x7fb6740a8390 con 0x7fb674138d80 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient(hunting): my global_id is 116022 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 -- 172.21.15.64:0/1364235811 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb67409db80 con 0x7fb674138d80 2024-09-17T15:54:19.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 -- 172.21.15.64:0/1364235811 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2180068433 0 0) 0x7fb664003520 con 0x7fb67409cd20 2024-09-17T15:54:19.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 -- 172.21.15.64:0/1364235811 >> v1:172.21.15.100:6791/0 conn(0x7fb6740a5de0 legacy=0x7fb67413e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 -- 172.21.15.64:0/1364235811 >> v1:172.21.15.100:6790/0 conn(0x7fb674138d80 legacy=0x7fb6740a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient: _finish_hunting 0 2024-09-17T15:54:19.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 monclient: found mon.noname-i 2024-09-17T15:54:19.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:19.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.100:6792/0 2024-09-17T15:54:19.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 -- 172.21.15.64:0/1364235811 --> v1:172.21.15.100:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67409e140 con 0x7fb67409cd20 2024-09-17T15:54:19.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient: _finish_auth 0 2024-09-17T15:54:19.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient: _check_auth_tickets 2024-09-17T15:54:19.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:19.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 -- 172.21.15.64:0/1364235811 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb66c001ae0 con 0x7fb67409cd20 2024-09-17T15:54:19.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-17T15:54:19.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient: dump: 2024-09-17T15:54:19.391 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.391 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.391 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.391 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.391 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.391 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb67af0e640 5 monclient: authenticate success, global_id 174292 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb67af0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 -- 172.21.15.64:0/1364235811 <== mon.7 v1:172.21.15.100:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb66c001c80 con 0x7fb67409cd20 2024-09-17T15:54:19.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb67af0e640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb67af0e640 4 set_mon_vals no callback set 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 1 -- 172.21.15.64:0/1364235811 <== mon.7 v1:172.21.15.100:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb66c003990 con 0x7fb67409cd20 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.379+0000 7fb6737fe640 10 monclient: dump: 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.380+0000 7fb67af0e640 10 monclient: shutdown 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.380+0000 7fb67af0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.380+0000 7fb67af0e640 1 -- 172.21.15.64:0/1364235811 >> v1:172.21.15.100:6792/0 conn(0x7fb67409cd20 legacy=0x7fb674141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.380+0000 7fb67af0e640 1 -- 172.21.15.64:0/1364235811 shutdown_connections 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.380+0000 7fb67af0e640 1 -- 172.21.15.64:0/1364235811 >> 172.21.15.64:0/1364235811 conn(0x7fb67406dc70 msgr2=0x7fb6740abb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:19.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.380+0000 7fb67af0e640 1 -- 172.21.15.64:0/1364235811 shutdown_connections 2024-09-17T15:54:19.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.380+0000 7fb67af0e640 1 -- 172.21.15.64:0/1364235811 wait complete. 2024-09-17T15:54:19.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.380+0000 7fb67af0e640 10 monclient: build_initial_monmap 2024-09-17T15:54:19.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.380+0000 7fb67af0e640 10 monclient: monmap: 2024-09-17T15:54:19.395 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:19.395 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:19.395 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:19.395 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:19.395 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:19.395 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:19.395 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:19.395 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:19.396 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:19.396 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:19.396 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:19.396 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:19.396 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:19.396 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:19.396 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:19.396 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.381+0000 7fb67af0e640 1 Processor -- start 2024-09-17T15:54:19.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.381+0000 7fb67af0e640 1 -- start start 2024-09-17T15:54:19.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.381+0000 7fb67af0e640 10 monclient: init 2024-09-17T15:54:19.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.381+0000 7fb67af0e640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:19.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.381+0000 7fb67af0e640 10 monclient: _add_conns ranks=[1,7,8,6,3,4,0,5,2] 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.381+0000 7fb67af0e640 10 monclient(hunting): picked mon.noname-f con 0x7fb67409cd20 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.381+0000 7fb67af0e640 10 monclient(hunting): picked mon.noname-i con 0x7fb6740a5de0 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.381+0000 7fb67af0e640 10 monclient(hunting): picked mon.noname-e con 0x7fb674138d80 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.381+0000 7fb67af0e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb674132910 con 0x7fb67409cd20 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.381+0000 7fb67af0e640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6740a5930 con 0x7fb6740a5de0 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.381+0000 7fb67af0e640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6741305b0 con 0x7fb674138d80 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.381+0000 7fb67af0e640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.381+0000 7fb67af0e640 10 monclient(hunting): authenticate will time out at 1257.056977s 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6794ab640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fb674138d80 0x7fb6741d6060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51118/0 (socket says 172.21.15.64:51118) 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6794ab640 1 -- 172.21.15.64:0/3742271251 learned_addr learned my addr 172.21.15.64:0/3742271251 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 998762700 0 0) 0x7fb6741305b0 con 0x7fb674138d80 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): my global_id is 174248 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb658003520 con 0x7fb674138d80 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3701632843 0 0) 0x7fb6740a5930 con 0x7fb6740a5de0 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): my global_id is 174301 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6741305b0 con 0x7fb6740a5de0 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2367525443 0 0) 0x7fb674132910 con 0x7fb67409cd20 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): my global_id is 7012 2024-09-17T15:54:19.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6740a5930 con 0x7fb67409cd20 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1214090704 0 0) 0x7fb658003520 con 0x7fb674138d80 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb674132910 con 0x7fb674138d80 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb66c0019a0 con 0x7fb674138d80 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): dump: 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3493179462 0 0) 0x7fb6741305b0 con 0x7fb6740a5de0 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb658003520 con 0x7fb6740a5de0 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb660001b10 con 0x7fb6740a5de0 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:19.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): dump: 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.401 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2588860887 0 0) 0x7fb6740a5930 con 0x7fb67409cd20 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6741305b0 con 0x7fb67409cd20 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb668001780 con 0x7fb67409cd20 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient(hunting): dump: 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.403 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.403 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.403 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.403 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.403 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.403 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.403 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.403 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.403 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.403 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2721587836 0 0) 0x7fb674132910 con 0x7fb674138d80 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 >> v1:172.21.15.100:6792/0 conn(0x7fb6740a5de0 legacy=0x7fb67412f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 >> v1:172.21.15.100:6789/0 conn(0x7fb67409cd20 legacy=0x7fb67412eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient: _finish_hunting 0 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 monclient: found mon.e 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67412f8e0 con 0x7fb674138d80 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient: _finish_auth 0 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient: _check_auth_tickets 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb67af0e640 5 monclient: authenticate success, global_id 174248 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb67af0e640 10 monclient: _renew_subs 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb67af0e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:19.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb67af0e640 1 -- 172.21.15.64:0/3742271251 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb674132070 con 0x7fb674138d80 2024-09-17T15:54:19.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb67af0e640 10 monclient: _renew_subs 2024-09-17T15:54:19.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb67af0e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:19.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb67af0e640 1 -- 172.21.15.64:0/3742271251 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fb674131380 con 0x7fb674138d80 2024-09-17T15:54:19.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb66c001bc0 con 0x7fb674138d80 2024-09-17T15:54:19.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:19.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb66c004240 con 0x7fb674138d80 2024-09-17T15:54:19.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb6717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.382+0000 7fb672ffd640 4 set_mon_vals no callback set 2024-09-17T15:54:19.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.383+0000 7fb6717fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.383+0000 7fb6717fa640 10 monclient: dump: 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.406 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.407 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.407 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.407 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.383+0000 7fb67af0e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:19.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.383+0000 7fb67af0e640 1 -- 172.21.15.64:0/3742271251 --> v1:172.21.15.64:6793/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fb654000f80 con 0x7fb674138d80 2024-09-17T15:54:19.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.383+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fb66c004460 con 0x7fb674138d80 2024-09-17T15:54:19.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.384+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7fb66c04ed90 con 0x7fb674138d80 2024-09-17T15:54:19.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.384+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 --> v1:172.21.15.100:6804/2791828047 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb65804b4f0 con 0x7fb658047880 2024-09-17T15:54:19.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.384+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== mon.8 v1:172.21.15.64:6793/0 9 ==== mon_get_version_reply(handle=1 version=327) ==== 24+0+0 (unknown 900077626 0 0) 0x7fb66c04f260 con 0x7fb674138d80 2024-09-17T15:54:19.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.384+0000 7fb6717fa640 10 monclient: handle_get_version_reply finishing 1 version 327 2024-09-17T15:54:19.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.384+0000 7fb678caa640 10 monclient: get_auth_request con 0x7fb658040c80 auth_method 0 2024-09-17T15:54:19.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.384+0000 7fb673fff640 10 monclient: get_auth_request con 0x7fb658047880 auth_method 0 2024-09-17T15:54:19.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.385+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== osd.3 v1:172.21.15.100:6804/2791828047 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb65804b4f0 con 0x7fb658047880 2024-09-17T15:54:19.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.397+0000 7fb67af0e640 1 -- 172.21.15.64:0/3742271251 --> v1:172.21.15.100:6804/2791828047 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb654002cd0 con 0x7fb658047880 2024-09-17T15:54:19.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.398+0000 7fb6717fa640 1 -- 172.21.15.64:0/3742271251 <== osd.3 v1:172.21.15.100:6804/2791828047 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fb654002cd0 con 0x7fb658047880 2024-09-17T15:54:19.408 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T15:54:19.408 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:19.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.398+0000 7fb67af0e640 1 -- 172.21.15.64:0/3742271251 >> v1:172.21.15.100:6804/2791828047 conn(0x7fb658047880 legacy=0x7fb658049ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.398+0000 7fb67af0e640 1 -- 172.21.15.64:0/3742271251 >> v1:172.21.15.100:6812/4210963484 conn(0x7fb658040c80 legacy=0x7fb658043140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.398+0000 7fb67af0e640 10 monclient: shutdown 2024-09-17T15:54:19.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.398+0000 7fb67af0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:19.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.398+0000 7fb67af0e640 1 -- 172.21.15.64:0/3742271251 >> v1:172.21.15.64:6793/0 conn(0x7fb674138d80 legacy=0x7fb6741d6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.398+0000 7fb6794ab640 1 -- 172.21.15.64:0/3742271251 reap_dead start 2024-09-17T15:54:19.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.398+0000 7fb67af0e640 1 -- 172.21.15.64:0/3742271251 shutdown_connections 2024-09-17T15:54:19.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.398+0000 7fb67af0e640 1 -- 172.21.15.64:0/3742271251 >> 172.21.15.64:0/3742271251 conn(0x7fb67406dc70 msgr2=0x7fb674135790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:19.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.398+0000 7fb67af0e640 1 -- 172.21.15.64:0/3742271251 shutdown_connections 2024-09-17T15:54:19.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.398+0000 7fb67af0e640 1 -- 172.21.15.64:0/3742271251 wait complete. 2024-09-17T15:54:19.411 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-17T15:54:19.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.512+0000 7fa9e1110640 10 monclient: get_monmap_and_config 2024-09-17T15:54:19.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.512+0000 7fa9e1110640 10 monclient: build_initial_monmap 2024-09-17T15:54:19.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.512+0000 7fa9e1110640 10 monclient: monmap: 2024-09-17T15:54:19.514 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:19.514 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.514 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:19.513490+0000 2024-09-17T15:54:19.515 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:19.513490+0000 2024-09-17T15:54:19.515 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:19.515 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.515 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:19.515 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:19.515 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:19.515 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:19.515 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:19.515 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:19.515 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:19.515 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:19.515 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:19.515 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.513+0000 7fa9e1110640 1 Processor -- start 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.513+0000 7fa9e1110640 1 -- start start 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.513+0000 7fa9e1110640 10 monclient: init 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.513+0000 7fa9e1110640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.513+0000 7fa9e1110640 10 monclient: _add_conns ranks=[4,3,2,0,8,6,5,1,7] 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.513+0000 7fa9e1110640 10 monclient(hunting): picked mon.noname-c con 0x7fa9dc0a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.513+0000 7fa9e1110640 10 monclient(hunting): picked mon.noname-g con 0x7fa9dc135b50 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.513+0000 7fa9e1110640 10 monclient(hunting): picked mon.noname-b con 0x7fa9dc135f70 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.513+0000 7fa9e1110640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9dc09d6f0 con 0x7fa9dc135f70 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.513+0000 7fa9e1110640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9dc09d9d0 con 0x7fa9dc135b50 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.513+0000 7fa9e1110640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9dc09dcb0 con 0x7fa9dc0a5520 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.513+0000 7fa9e1110640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.513+0000 7fa9e1110640 10 monclient(hunting): authenticate will time out at 1257.188958s 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9dad9d640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fa9dc0a5520 0x7fa9dc0a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:59872/0 (socket says 172.21.15.64:59872) 2024-09-17T15:54:19.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9dad9d640 1 -- 172.21.15.64:0/3447778972 learned_addr learned my addr 172.21.15.64:0/3447778972 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 -- 172.21.15.64:0/3447778972 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2513904089 0 0) 0x7fa9dc09dcb0 con 0x7fa9dc0a5520 2024-09-17T15:54:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient(hunting): my global_id is 154750 2024-09-17T15:54:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 -- 172.21.15.64:0/3447778972 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9c8003520 con 0x7fa9dc0a5520 2024-09-17T15:54:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 -- 172.21.15.64:0/3447778972 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 813452088 0 0) 0x7fa9dc09d6f0 con 0x7fa9dc135f70 2024-09-17T15:54:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient(hunting): my global_id is 125327 2024-09-17T15:54:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 -- 172.21.15.64:0/3447778972 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9dc09dcb0 con 0x7fa9dc135f70 2024-09-17T15:54:19.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 -- 172.21.15.64:0/3447778972 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1325775336 0 0) 0x7fa9dc09d9d0 con 0x7fa9dc135b50 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient(hunting): my global_id is 116031 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 -- 172.21.15.64:0/3447778972 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9dc09d6f0 con 0x7fa9dc135b50 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 -- 172.21.15.64:0/3447778972 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3376735132 0 0) 0x7fa9c8003520 con 0x7fa9dc0a5520 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 -- 172.21.15.64:0/3447778972 >> v1:172.21.15.100:6790/0 conn(0x7fa9dc135b50 legacy=0x7fa9dc13deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 -- 172.21.15.64:0/3447778972 >> v1:172.21.15.64:6790/0 conn(0x7fa9dc135f70 legacy=0x7fa9dc1415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient: _finish_hunting 0 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 monclient: found mon.noname-c 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 -- 172.21.15.64:0/3447778972 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9dc09ccd0 con 0x7fa9dc0a5520 2024-09-17T15:54:19.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient: _finish_auth 0 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient: _check_auth_tickets 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9e1110640 5 monclient: authenticate success, global_id 154750 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 -- 172.21.15.64:0/3447778972 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa9cc001760 con 0x7fa9dc0a5520 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9e1110640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient: dump: 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.526 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9e1110640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 -- 172.21.15.64:0/3447778972 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa9cc001980 con 0x7fa9dc0a5520 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:19.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 1 -- 172.21.15.64:0/3447778972 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa9cc004be0 con 0x7fa9dc0a5520 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9da59c640 10 monclient: dump: 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.528 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9e1110640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.514+0000 7fa9e1110640 4 set_mon_vals no callback set 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.515+0000 7fa9e1110640 10 monclient: shutdown 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.515+0000 7fa9e1110640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.515+0000 7fa9e1110640 1 -- 172.21.15.64:0/3447778972 >> v1:172.21.15.64:6791/0 conn(0x7fa9dc0a5520 legacy=0x7fa9dc0a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.515+0000 7fa9e1110640 1 -- 172.21.15.64:0/3447778972 shutdown_connections 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.515+0000 7fa9e1110640 1 -- 172.21.15.64:0/3447778972 >> 172.21.15.64:0/3447778972 conn(0x7fa9dc06db30 msgr2=0x7fa9dc0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.515+0000 7fa9e1110640 1 -- 172.21.15.64:0/3447778972 shutdown_connections 2024-09-17T15:54:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.515+0000 7fa9e1110640 1 -- 172.21.15.64:0/3447778972 wait complete. 2024-09-17T15:54:19.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 10 monclient: build_initial_monmap 2024-09-17T15:54:19.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 10 monclient: monmap: 2024-09-17T15:54:19.530 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:19.530 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:19.530 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:19.530 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:19.530 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:19.530 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:19.530 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:19.530 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:19.530 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:19.531 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:19.531 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:19.531 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:19.531 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:19.531 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:19.531 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:19.531 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 1 Processor -- start 2024-09-17T15:54:19.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 1 -- start start 2024-09-17T15:54:19.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 10 monclient: init 2024-09-17T15:54:19.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:19.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 10 monclient: _add_conns ranks=[5,4,6,1,3,8,0,7,2] 2024-09-17T15:54:19.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 10 monclient(hunting): picked mon.noname-h con 0x7fa9dc0a5520 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:19.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 10 monclient(hunting): picked mon.noname-c con 0x7fa9dc135b50 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:19.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 10 monclient(hunting): picked mon.noname-d con 0x7fa9dc135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:19.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9dc09f580 con 0x7fa9dc135b50 2024-09-17T15:54:19.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9dc09f290 con 0x7fa9dc0a5520 2024-09-17T15:54:19.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9dc132740 con 0x7fa9dc135f70 2024-09-17T15:54:19.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:19.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.516+0000 7fa9e1110640 10 monclient(hunting): authenticate will time out at 1257.191958s 2024-09-17T15:54:19.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d3fff640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fa9dc135b50 0x7fa9dc09eb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:59876/0 (socket says 172.21.15.64:59876) 2024-09-17T15:54:19.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d3fff640 1 -- 172.21.15.64:0/2264895738 learned_addr learned my addr 172.21.15.64:0/2264895738 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:19.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9db59e640 1 --1- 172.21.15.64:0/2264895738 >> v1:172.21.15.64:6792/0 conn(0x7fa9dc135f70 0x7fa9dc1d6360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:54886/0 (socket says 172.21.15.64:54886) 2024-09-17T15:54:19.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3946974150 0 0) 0x7fa9dc09f580 con 0x7fa9dc135b50 2024-09-17T15:54:19.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): my global_id is 154759 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9bc003520 con 0x7fa9dc135b50 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2200472947 0 0) 0x7fa9dc132740 con 0x7fa9dc135f70 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): my global_id is 174264 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9dc09f580 con 0x7fa9dc135f70 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3450561422 0 0) 0x7fa9dc09f290 con 0x7fa9dc0a5520 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): my global_id is 174326 2024-09-17T15:54:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9dc132740 con 0x7fa9dc0a5520 2024-09-17T15:54:19.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2087502987 0 0) 0x7fa9bc003520 con 0x7fa9dc135b50 2024-09-17T15:54:19.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9dc09f290 con 0x7fa9dc135b50 2024-09-17T15:54:19.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa9c4001bb0 con 0x7fa9dc135b50 2024-09-17T15:54:19.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:19.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:19.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): dump: 2024-09-17T15:54:19.534 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.534 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.534 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.534 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.534 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.535 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.535 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.535 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.535 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.535 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.535 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.535 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.535 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.535 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.535 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.535 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:19.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4212874839 0 0) 0x7fa9dc09f580 con 0x7fa9dc135f70 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9bc003520 con 0x7fa9dc135f70 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa9d4001bd0 con 0x7fa9dc135f70 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): dump: 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.536 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2310724800 0 0) 0x7fa9dc132740 con 0x7fa9dc0a5520 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9dc09f580 con 0x7fa9dc0a5520 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa9cc001720 con 0x7fa9dc0a5520 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient(hunting): dump: 2024-09-17T15:54:19.537 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.538 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.539 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2605083339 0 0) 0x7fa9dc09f290 con 0x7fa9dc135b50 2024-09-17T15:54:19.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 >> v1:172.21.15.64:6792/0 conn(0x7fa9dc135f70 legacy=0x7fa9dc1d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 >> v1:172.21.15.100:6791/0 conn(0x7fa9dc0a5520 legacy=0x7fa9dc09e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:54:19.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 monclient: found mon.c 2024-09-17T15:54:19.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:19.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:19.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9dc09faf0 con 0x7fa9dc135b50 2024-09-17T15:54:19.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient: _finish_auth 0 2024-09-17T15:54:19.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:54:19.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:19.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9e1110640 5 monclient: authenticate success, global_id 154759 2024-09-17T15:54:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9e1110640 10 monclient: _renew_subs 2024-09-17T15:54:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9e1110640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9e1110640 1 -- 172.21.15.64:0/2264895738 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa9dc0a0760 con 0x7fa9dc135b50 2024-09-17T15:54:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9e1110640 10 monclient: _renew_subs 2024-09-17T15:54:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9e1110640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.517+0000 7fa9e1110640 1 -- 172.21.15.64:0/2264895738 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9dc0a51f0 con 0x7fa9dc135b50 2024-09-17T15:54:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.518+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa9c40038c0 con 0x7fa9dc135b50 2024-09-17T15:54:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.518+0000 7fa9d37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.518+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fa9c4003e30 con 0x7fa9dc135b50 2024-09-17T15:54:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.518+0000 7fa9d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.518+0000 7fa9d9d9b640 4 set_mon_vals no callback set 2024-09-17T15:54:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.518+0000 7fa9d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.518+0000 7fa9d37fe640 10 monclient: dump: 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.541 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.542 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.542 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.519+0000 7fa9e1110640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:19.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.519+0000 7fa9e1110640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:19.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.519+0000 7fa9e1110640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:19.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.519+0000 7fa9e1110640 1 -- 172.21.15.64:0/2264895738 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9c0005080 con 0x7fa9dc135b50 2024-09-17T15:54:19.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.519+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fa9c4003fd0 con 0x7fa9dc135b50 2024-09-17T15:54:19.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.519+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7fa9c404ec50 con 0x7fa9dc135b50 2024-09-17T15:54:19.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.519+0000 7fa9dad9d640 10 monclient: get_auth_request con 0x7fa9bc040bd0 auth_method 0 2024-09-17T15:54:19.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.523+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fa9c40178a0 con 0x7fa9dc135b50 2024-09-17T15:54:19.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.523+0000 7fa9d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:19.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.523+0000 7fa9d37fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:19.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.707+0000 7fa9e1110640 1 -- 172.21.15.64:0/2264895738 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fa9c0000b90 con 0x7fa9bc040bd0 2024-09-17T15:54:19.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.707+0000 7fa9d37fe640 1 -- 172.21.15.64:0/2264895738 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (unknown 3526570807 0 0) 0x7fa9c0000b90 con 0x7fa9bc040bd0 2024-09-17T15:54:19.709 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 3 to deep-scrub 2024-09-17T15:54:19.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.709+0000 7fa9e1110640 1 -- 172.21.15.64:0/2264895738 >> v1:172.21.15.100:6812/4210963484 conn(0x7fa9bc040bd0 legacy=0x7fa9bc043090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.709+0000 7fa9e1110640 10 monclient: shutdown 2024-09-17T15:54:19.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.709+0000 7fa9e1110640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:19.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.709+0000 7fa9e1110640 1 -- 172.21.15.64:0/2264895738 >> v1:172.21.15.64:6791/0 conn(0x7fa9dc135b50 legacy=0x7fa9dc09eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.709+0000 7fa9e1110640 1 -- 172.21.15.64:0/2264895738 shutdown_connections 2024-09-17T15:54:19.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.709+0000 7fa9e1110640 1 -- 172.21.15.64:0/2264895738 >> 172.21.15.64:0/2264895738 conn(0x7fa9dc06db30 msgr2=0x7fa9dc1334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:19.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.709+0000 7fa9e1110640 1 -- 172.21.15.64:0/2264895738 shutdown_connections 2024-09-17T15:54:19.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.709+0000 7fa9e1110640 1 -- 172.21.15.64:0/2264895738 wait complete. 2024-09-17T15:54:19.722 INFO:tasks.ceph:Scrubbing osd.4 2024-09-17T15:54:19.722 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-17T15:54:19.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.825+0000 7fd18c9bb640 10 monclient: get_monmap_and_config 2024-09-17T15:54:19.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.825+0000 7fd18c9bb640 10 monclient: build_initial_monmap 2024-09-17T15:54:19.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.825+0000 7fd18c9bb640 10 monclient: monmap: 2024-09-17T15:54:19.827 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:19.827 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.827 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:19.826318+0000 2024-09-17T15:54:19.827 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:19.826318+0000 2024-09-17T15:54:19.827 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:19.828 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.828 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:19.828 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:19.828 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:19.828 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:19.828 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:19.828 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:19.828 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:19.829 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:19.829 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:19.829 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18c9bb640 1 Processor -- start 2024-09-17T15:54:19.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18c9bb640 1 -- start start 2024-09-17T15:54:19.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18c9bb640 10 monclient: init 2024-09-17T15:54:19.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18c9bb640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:19.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18c9bb640 10 monclient: _add_conns ranks=[8,1,6,2,0,5,7,4,3] 2024-09-17T15:54:19.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18c9bb640 10 monclient(hunting): picked mon.noname-e con 0x7fd188138d80 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:19.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18c9bb640 10 monclient(hunting): picked mon.noname-f con 0x7fd1880a5de0 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:19.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18c9bb640 10 monclient(hunting): picked mon.noname-d con 0x7fd18809cd20 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:19.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18c9bb640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1880a8390 con 0x7fd1880a5de0 2024-09-17T15:54:19.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18c9bb640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd18809db80 con 0x7fd18809cd20 2024-09-17T15:54:19.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18c9bb640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd18809de60 con 0x7fd188138d80 2024-09-17T15:54:19.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18c9bb640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:19.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18c9bb640 10 monclient(hunting): authenticate will time out at 1257.501642s 2024-09-17T15:54:19.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18659c640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fd188138d80 0x7fd1880a56d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51126/0 (socket says 172.21.15.64:51126) 2024-09-17T15:54:19.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.826+0000 7fd18659c640 1 -- 172.21.15.64:0/2133573934 learned_addr learned my addr 172.21.15.64:0/2133573934 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:19.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 1 -- 172.21.15.64:0/2133573934 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3288528344 0 0) 0x7fd18809de60 con 0x7fd188138d80 2024-09-17T15:54:19.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient(hunting): my global_id is 174257 2024-09-17T15:54:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 1 -- 172.21.15.64:0/2133573934 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd174003520 con 0x7fd188138d80 2024-09-17T15:54:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 1 -- 172.21.15.64:0/2133573934 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2682119409 0 0) 0x7fd18809db80 con 0x7fd18809cd20 2024-09-17T15:54:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient(hunting): my global_id is 174273 2024-09-17T15:54:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 1 -- 172.21.15.64:0/2133573934 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd18809de60 con 0x7fd18809cd20 2024-09-17T15:54:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 1 -- 172.21.15.64:0/2133573934 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1158385278 0 0) 0x7fd174003520 con 0x7fd188138d80 2024-09-17T15:54:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 1 -- 172.21.15.64:0/2133573934 >> v1:172.21.15.64:6792/0 conn(0x7fd18809cd20 legacy=0x7fd188141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 1 -- 172.21.15.64:0/2133573934 >> v1:172.21.15.100:6789/0 conn(0x7fd1880a5de0 legacy=0x7fd18813e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:54:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 1 monclient: found mon.noname-e 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.64:6793/0 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 1 -- 172.21.15.64:0/2133573934 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd18809e140 con 0x7fd188138d80 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient: _finish_auth 0 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 1 -- 172.21.15.64:0/2133573934 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd178001760 con 0x7fd188138d80 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd18c9bb640 5 monclient: authenticate success, global_id 174257 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd18c9bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient: dump: 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.842 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.842 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.842 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.842 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.842 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.842 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.842 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.842 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.842 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.843 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.843 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.843 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.843 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.843 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd18c9bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:19.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 1 -- 172.21.15.64:0/2133573934 <== mon.8 v1:172.21.15.64:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd178001900 con 0x7fd188138d80 2024-09-17T15:54:19.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:19.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 1 -- 172.21.15.64:0/2133573934 <== mon.8 v1:172.21.15.64:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd1780048a0 con 0x7fd188138d80 2024-09-17T15:54:19.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd18c9bb640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:19.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd18c9bb640 4 set_mon_vals no callback set 2024-09-17T15:54:19.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:19.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd185d9b640 10 monclient: dump: 2024-09-17T15:54:19.844 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.844 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.845 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.845 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.845 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.845 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.845 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.845 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.846 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.846 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.846 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.846 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.846 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.846 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.846 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.847 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd18c9bb640 10 monclient: shutdown 2024-09-17T15:54:19.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd18c9bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:19.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd18c9bb640 1 -- 172.21.15.64:0/2133573934 >> v1:172.21.15.64:6793/0 conn(0x7fd188138d80 legacy=0x7fd1880a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd18c9bb640 1 -- 172.21.15.64:0/2133573934 shutdown_connections 2024-09-17T15:54:19.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.827+0000 7fd18c9bb640 1 -- 172.21.15.64:0/2133573934 >> 172.21.15.64:0/2133573934 conn(0x7fd18806dc70 msgr2=0x7fd1880abb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:19.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.828+0000 7fd18c9bb640 1 -- 172.21.15.64:0/2133573934 shutdown_connections 2024-09-17T15:54:19.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.828+0000 7fd18c9bb640 1 -- 172.21.15.64:0/2133573934 wait complete. 2024-09-17T15:54:19.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.828+0000 7fd18c9bb640 10 monclient: build_initial_monmap 2024-09-17T15:54:19.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.828+0000 7fd18c9bb640 10 monclient: monmap: 2024-09-17T15:54:19.848 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:19.848 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:19.848 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:19.848 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:19.848 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:19.848 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:19.849 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:19.849 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:19.849 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:19.849 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:19.849 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:19.849 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:19.849 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:19.849 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:19.850 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:19.850 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd18c9bb640 1 Processor -- start 2024-09-17T15:54:19.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd18c9bb640 1 -- start start 2024-09-17T15:54:19.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd18c9bb640 10 monclient: init 2024-09-17T15:54:19.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd18c9bb640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:19.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd18c9bb640 10 monclient: _add_conns ranks=[7,4,3,8,5,2,1,6,0] 2024-09-17T15:54:19.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd18c9bb640 10 monclient(hunting): picked mon.noname-i con 0x7fd18809cd20 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:19.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd18c9bb640 10 monclient(hunting): picked mon.noname-c con 0x7fd1880a5de0 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:19.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd18c9bb640 10 monclient(hunting): picked mon.noname-g con 0x7fd188138d80 addr v1:172.21.15.100:6790/0 2024-09-17T15:54:19.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd18c9bb640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd188132740 con 0x7fd188138d80 2024-09-17T15:54:19.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd18c9bb640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1880a59d0 con 0x7fd1880a5de0 2024-09-17T15:54:19.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd18c9bb640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1881303e0 con 0x7fd18809cd20 2024-09-17T15:54:19.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd18c9bb640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:19.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd18c9bb640 10 monclient(hunting): authenticate will time out at 1257.504887s 2024-09-17T15:54:19.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd17dd9b640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fd1880a5de0 0x7fd18812f040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:59888/0 (socket says 172.21.15.64:59888) 2024-09-17T15:54:19.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.829+0000 7fd17dd9b640 1 -- 172.21.15.64:0/629991453 learned_addr learned my addr 172.21.15.64:0/629991453 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:19.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 874013510 0 0) 0x7fd1880a59d0 con 0x7fd1880a5de0 2024-09-17T15:54:19.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): my global_id is 154768 2024-09-17T15:54:19.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd168003520 con 0x7fd1880a5de0 2024-09-17T15:54:19.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 184604294 0 0) 0x7fd1881303e0 con 0x7fd18809cd20 2024-09-17T15:54:19.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): my global_id is 174310 2024-09-17T15:54:19.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1880a59d0 con 0x7fd18809cd20 2024-09-17T15:54:19.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1774253120 0 0) 0x7fd168003520 con 0x7fd1880a5de0 2024-09-17T15:54:19.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1881303e0 con 0x7fd1880a5de0 2024-09-17T15:54:19.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd170001b90 con 0x7fd1880a5de0 2024-09-17T15:54:19.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:19.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): dump: 2024-09-17T15:54:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.855 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.855 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.855 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.855 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:19.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 <== mon.3 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2658500236 0 0) 0x7fd188132740 con 0x7fd188138d80 2024-09-17T15:54:19.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient(hunting): my global_id is 116040 2024-09-17T15:54:19.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd168003520 con 0x7fd188138d80 2024-09-17T15:54:19.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2790114596 0 0) 0x7fd1881303e0 con 0x7fd1880a5de0 2024-09-17T15:54:19.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 >> v1:172.21.15.100:6792/0 conn(0x7fd18809cd20 legacy=0x7fd18812e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 >> v1:172.21.15.100:6790/0 conn(0x7fd188138d80 legacy=0x7fd1881d6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:54:19.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 monclient: found mon.c 2024-09-17T15:54:19.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:19.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:19.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd188132460 con 0x7fd1880a5de0 2024-09-17T15:54:19.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:54:19.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:54:19.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:19.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd18c9bb640 5 monclient: authenticate success, global_id 154768 2024-09-17T15:54:19.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd18c9bb640 10 monclient: _renew_subs 2024-09-17T15:54:19.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd18c9bb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:19.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd18c9bb640 1 -- 172.21.15.64:0/629991453 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1881312f0 con 0x7fd1880a5de0 2024-09-17T15:54:19.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd18c9bb640 10 monclient: _renew_subs 2024-09-17T15:54:19.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd18c9bb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:19.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd18c9bb640 1 -- 172.21.15.64:0/629991453 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fd188132ed0 con 0x7fd1880a5de0 2024-09-17T15:54:19.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1700038c0 con 0x7fd1880a5de0 2024-09-17T15:54:19.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:19.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd170003df0 con 0x7fd1880a5de0 2024-09-17T15:54:19.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:19.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd17f7fe640 10 monclient: dump: 2024-09-17T15:54:19.861 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.861 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.861 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.861 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.861 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.862 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.862 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.862 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.862 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.862 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.862 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.862 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.863 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.863 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.863 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.863 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.830+0000 7fd184d99640 4 set_mon_vals no callback set 2024-09-17T15:54:19.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.831+0000 7fd18c9bb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:19.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.831+0000 7fd18c9bb640 1 -- 172.21.15.64:0/629991453 --> v1:172.21.15.64:6791/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fd1880a9b70 con 0x7fd1880a5de0 2024-09-17T15:54:19.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.831+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fd170003f90 con 0x7fd1880a5de0 2024-09-17T15:54:19.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.831+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7fd17004ea30 con 0x7fd1880a5de0 2024-09-17T15:54:19.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.832+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 --> v1:172.21.15.100:6808/4242749853 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd16804aa70 con 0x7fd168046e00 2024-09-17T15:54:19.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.832+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_get_version_reply(handle=1 version=327) ==== 24+0+0 (unknown 900077626 0 0) 0x7fd17004ef00 con 0x7fd1880a5de0 2024-09-17T15:54:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.832+0000 7fd17f7fe640 10 monclient: handle_get_version_reply finishing 1 version 327 2024-09-17T15:54:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.832+0000 7fd18659c640 10 monclient: get_auth_request con 0x7fd168040220 auth_method 0 2024-09-17T15:54:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.832+0000 7fd186d9d640 10 monclient: get_auth_request con 0x7fd168046e00 auth_method 0 2024-09-17T15:54:19.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.833+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 <== osd.4 v1:172.21.15.100:6808/4242749853 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd16804aa70 con 0x7fd168046e00 2024-09-17T15:54:19.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.845+0000 7fd18c9bb640 1 -- 172.21.15.64:0/629991453 --> v1:172.21.15.100:6808/4242749853 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd188130f40 con 0x7fd168046e00 2024-09-17T15:54:19.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.846+0000 7fd17f7fe640 1 -- 172.21.15.64:0/629991453 <== osd.4 v1:172.21.15.100:6808/4242749853 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fd188130f40 con 0x7fd168046e00 2024-09-17T15:54:19.867 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T15:54:19.867 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:19.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.846+0000 7fd18c9bb640 1 -- 172.21.15.64:0/629991453 >> v1:172.21.15.100:6808/4242749853 conn(0x7fd168046e00 legacy=0x7fd168049220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.846+0000 7fd18c9bb640 1 -- 172.21.15.64:0/629991453 >> v1:172.21.15.100:6812/4210963484 conn(0x7fd168040220 legacy=0x7fd1680426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.846+0000 7fd18c9bb640 10 monclient: shutdown 2024-09-17T15:54:19.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.846+0000 7fd18c9bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:19.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.846+0000 7fd18c9bb640 1 -- 172.21.15.64:0/629991453 >> v1:172.21.15.64:6791/0 conn(0x7fd1880a5de0 legacy=0x7fd18812f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.846+0000 7fd186d9d640 1 -- 172.21.15.64:0/629991453 reap_dead start 2024-09-17T15:54:19.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.846+0000 7fd18c9bb640 1 -- 172.21.15.64:0/629991453 shutdown_connections 2024-09-17T15:54:19.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.846+0000 7fd18c9bb640 1 -- 172.21.15.64:0/629991453 >> 172.21.15.64:0/629991453 conn(0x7fd18806dc70 msgr2=0x7fd1880ab2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:19.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.846+0000 7fd18c9bb640 1 -- 172.21.15.64:0/629991453 shutdown_connections 2024-09-17T15:54:19.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.846+0000 7fd18c9bb640 1 -- 172.21.15.64:0/629991453 wait complete. 2024-09-17T15:54:19.872 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-17T15:54:19.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.973+0000 7fd641672640 10 monclient: get_monmap_and_config 2024-09-17T15:54:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.973+0000 7fd641672640 10 monclient: build_initial_monmap 2024-09-17T15:54:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.973+0000 7fd641672640 10 monclient: monmap: 2024-09-17T15:54:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:19.973916+0000 2024-09-17T15:54:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:19.973916+0000 2024-09-17T15:54:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:19.976 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:19.976 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:19.976 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:19.976 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:19.976 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.973+0000 7fd641672640 1 Processor -- start 2024-09-17T15:54:19.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.973+0000 7fd641672640 1 -- start start 2024-09-17T15:54:19.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.973+0000 7fd641672640 10 monclient: init 2024-09-17T15:54:19.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd641672640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:19.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd641672640 10 monclient: _add_conns ranks=[4,0,5,8,6,7,3,1,2] 2024-09-17T15:54:19.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd641672640 10 monclient(hunting): picked mon.noname-c con 0x7fd63c0a54b0 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:19.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd641672640 10 monclient(hunting): picked mon.noname-a con 0x7fd63c135c80 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:19.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd641672640 10 monclient(hunting): picked mon.noname-h con 0x7fd63c1360a0 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:19.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd641672640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63c0aacb0 con 0x7fd63c135c80 2024-09-17T15:54:19.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd641672640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63c09d780 con 0x7fd63c0a54b0 2024-09-17T15:54:19.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd641672640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63c09da60 con 0x7fd63c1360a0 2024-09-17T15:54:19.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd641672640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:19.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd641672640 10 monclient(hunting): authenticate will time out at 1257.649237s 2024-09-17T15:54:19.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd6327fc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fd63c135c80 0x7fd63c13df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35342/0 (socket says 172.21.15.64:35342) 2024-09-17T15:54:19.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd6327fc640 1 -- 172.21.15.64:0/9820102 learned_addr learned my addr 172.21.15.64:0/9820102 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:19.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63affd640 1 --1- 172.21.15.64:0/9820102 >> v1:172.21.15.64:6791/0 conn(0x7fd63c0a54b0 0x7fd63c0a5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:59902/0 (socket says 172.21.15.64:59902) 2024-09-17T15:54:19.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 1 -- 172.21.15.64:0/9820102 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2849841593 0 0) 0x7fd63c0aacb0 con 0x7fd63c135c80 2024-09-17T15:54:19.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient(hunting): my global_id is 125208 2024-09-17T15:54:19.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 1 -- 172.21.15.64:0/9820102 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd628003520 con 0x7fd63c135c80 2024-09-17T15:54:19.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 1 -- 172.21.15.64:0/9820102 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 711651134 0 0) 0x7fd63c09d780 con 0x7fd63c0a54b0 2024-09-17T15:54:19.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient(hunting): my global_id is 154777 2024-09-17T15:54:19.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 1 -- 172.21.15.64:0/9820102 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd63c0aacb0 con 0x7fd63c0a54b0 2024-09-17T15:54:19.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 1 -- 172.21.15.64:0/9820102 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 446313436 0 0) 0x7fd63c09da60 con 0x7fd63c1360a0 2024-09-17T15:54:19.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient(hunting): my global_id is 174335 2024-09-17T15:54:19.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 1 -- 172.21.15.64:0/9820102 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd63c09d780 con 0x7fd63c1360a0 2024-09-17T15:54:19.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 1 -- 172.21.15.64:0/9820102 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 561108748 0 0) 0x7fd628003520 con 0x7fd63c135c80 2024-09-17T15:54:19.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 1 -- 172.21.15.64:0/9820102 >> v1:172.21.15.100:6791/0 conn(0x7fd63c1360a0 legacy=0x7fd63c141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 1 -- 172.21.15.64:0/9820102 >> v1:172.21.15.64:6791/0 conn(0x7fd63c0a54b0 legacy=0x7fd63c0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 1 monclient: found mon.noname-a 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 1 -- 172.21.15.64:0/9820102 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63c09dd40 con 0x7fd63c135c80 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 1 -- 172.21.15.64:0/9820102 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd624001830 con 0x7fd63c135c80 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd641672640 5 monclient: authenticate success, global_id 125208 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd641672640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:54:19.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd63a7fc640 10 monclient: dump: 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.987 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.988 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.988 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.974+0000 7fd641672640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:19.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd63a7fc640 1 -- 172.21.15.64:0/9820102 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd6240019d0 con 0x7fd63c135c80 2024-09-17T15:54:19.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd63a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:19.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd63a7fc640 1 -- 172.21.15.64:0/9820102 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd624004be0 con 0x7fd63c135c80 2024-09-17T15:54:19.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd641672640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:19.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd641672640 4 set_mon_vals no callback set 2024-09-17T15:54:19.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd63a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd63a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:19.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd63a7fc640 10 monclient: dump: 2024-09-17T15:54:19.988 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.988 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.989 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.989 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.989 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.989 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.989 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.989 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.989 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.989 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.989 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.989 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.989 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.989 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.989 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.990 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd641672640 10 monclient: shutdown 2024-09-17T15:54:19.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd641672640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:19.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd641672640 1 -- 172.21.15.64:0/9820102 >> v1:172.21.15.64:6789/0 conn(0x7fd63c135c80 legacy=0x7fd63c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd641672640 1 -- 172.21.15.64:0/9820102 shutdown_connections 2024-09-17T15:54:19.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd641672640 1 -- 172.21.15.64:0/9820102 >> 172.21.15.64:0/9820102 conn(0x7fd63c06daa0 msgr2=0x7fd63c0ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:19.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd641672640 1 -- 172.21.15.64:0/9820102 shutdown_connections 2024-09-17T15:54:19.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.975+0000 7fd641672640 1 -- 172.21.15.64:0/9820102 wait complete. 2024-09-17T15:54:19.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.976+0000 7fd641672640 10 monclient: build_initial_monmap 2024-09-17T15:54:19.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.976+0000 7fd641672640 10 monclient: monmap: 2024-09-17T15:54:19.990 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:19.990 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:19.990 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.976+0000 7fd641672640 1 Processor -- start 2024-09-17T15:54:19.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.976+0000 7fd641672640 1 -- start start 2024-09-17T15:54:19.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.976+0000 7fd641672640 10 monclient: init 2024-09-17T15:54:19.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:19.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 10 monclient: _add_conns ranks=[5,8,7,6,4,3,1,2,0] 2024-09-17T15:54:19.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 10 monclient(hunting): picked mon.noname-h con 0x7fd63c0a54b0 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:19.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 10 monclient(hunting): picked mon.noname-e con 0x7fd63c135c80 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:19.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 10 monclient(hunting): picked mon.noname-i con 0x7fd63c1360a0 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:19.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63c0a07f0 con 0x7fd63c0a54b0 2024-09-17T15:54:19.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63c0a0ad0 con 0x7fd63c1360a0 2024-09-17T15:54:19.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63c09f120 con 0x7fd63c135c80 2024-09-17T15:54:19.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:19.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 10 monclient(hunting): authenticate will time out at 1257.652125s 2024-09-17T15:54:19.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd6327fc640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7fd63c135c80 0x7fd63c09ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51128/0 (socket says 172.21.15.64:51128) 2024-09-17T15:54:19.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd6327fc640 1 -- 172.21.15.64:0/1770573309 learned_addr learned my addr 172.21.15.64:0/1770573309 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:19.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 502127183 0 0) 0x7fd63c09f120 con 0x7fd63c135c80 2024-09-17T15:54:19.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): my global_id is 174266 2024-09-17T15:54:19.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd61c003520 con 0x7fd63c135c80 2024-09-17T15:54:19.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3509885981 0 0) 0x7fd63c0a07f0 con 0x7fd63c0a54b0 2024-09-17T15:54:19.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): my global_id is 174344 2024-09-17T15:54:19.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd63c09f120 con 0x7fd63c0a54b0 2024-09-17T15:54:19.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3357476810 0 0) 0x7fd63c0a0ad0 con 0x7fd63c1360a0 2024-09-17T15:54:19.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:19.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:19.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:19.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): my global_id is 174319 2024-09-17T15:54:19.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd63c0a07f0 con 0x7fd63c1360a0 2024-09-17T15:54:19.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 775121153 0 0) 0x7fd61c003520 con 0x7fd63c135c80 2024-09-17T15:54:19.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd63c0a0ad0 con 0x7fd63c135c80 2024-09-17T15:54:19.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd6240017f0 con 0x7fd63c135c80 2024-09-17T15:54:19.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:19.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-17T15:54:19.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): dump: 2024-09-17T15:54:19.996 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.996 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:19.997 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1670017691 0 0) 0x7fd63c0a0ad0 con 0x7fd63c135c80 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 >> v1:172.21.15.100:6792/0 conn(0x7fd63c1360a0 legacy=0x7fd63c1d6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 >> v1:172.21.15.100:6791/0 conn(0x7fd63c0a54b0 legacy=0x7fd63c09e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient: _finish_hunting 0 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 monclient: found mon.e 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63c0a6fc0 con 0x7fd63c135c80 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient: _finish_auth 0 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 10 monclient: _check_auth_tickets 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd633fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 5 monclient: authenticate success, global_id 174266 2024-09-17T15:54:19.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 10 monclient: _renew_subs 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.977+0000 7fd641672640 1 -- 172.21.15.64:0/1770573309 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd63c0a0510 con 0x7fd63c135c80 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd641672640 10 monclient: _renew_subs 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd641672640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd641672640 1 -- 172.21.15.64:0/1770573309 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fd63c0a0ef0 con 0x7fd63c135c80 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd624001990 con 0x7fd63c135c80 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd633fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fd624004b50 con 0x7fd63c135c80 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd633fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd639ffb640 4 set_mon_vals no callback set 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd633fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd633fff640 10 monclient: dump: 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:19.999 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.000 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fd624011960 con 0x7fd63c135c80 2024-09-17T15:54:20.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd641672640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:20.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd641672640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:20.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd641672640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:20.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.978+0000 7fd641672640 1 -- 172.21.15.64:0/1770573309 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd620005080 con 0x7fd63c135c80 2024-09-17T15:54:20.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.979+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7fd62404ea50 con 0x7fd63c135c80 2024-09-17T15:54:20.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.979+0000 7fd63affd640 10 monclient: get_auth_request con 0x7fd61c040040 auth_method 0 2024-09-17T15:54:20.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.983+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 <== mon.8 v1:172.21.15.64:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fd6240177c0 con 0x7fd63c135c80 2024-09-17T15:54:20.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.983+0000 7fd633fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:20.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:19.983+0000 7fd633fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:20.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.173+0000 7fd641672640 1 -- 172.21.15.64:0/1770573309 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fd620000b90 con 0x7fd61c040040 2024-09-17T15:54:20.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.173+0000 7fd633fff640 1 -- 172.21.15.64:0/1770573309 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (unknown 3782719169 0 0) 0x7fd620000b90 con 0x7fd61c040040 2024-09-17T15:54:20.175 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 4 to deep-scrub 2024-09-17T15:54:20.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.175+0000 7fd641672640 1 -- 172.21.15.64:0/1770573309 >> v1:172.21.15.100:6812/4210963484 conn(0x7fd61c040040 legacy=0x7fd61c042500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.175+0000 7fd641672640 10 monclient: shutdown 2024-09-17T15:54:20.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.175+0000 7fd641672640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:20.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.175+0000 7fd641672640 1 -- 172.21.15.64:0/1770573309 >> v1:172.21.15.64:6793/0 conn(0x7fd63c135c80 legacy=0x7fd63c09ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.175+0000 7fd641672640 1 -- 172.21.15.64:0/1770573309 shutdown_connections 2024-09-17T15:54:20.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.175+0000 7fd641672640 1 -- 172.21.15.64:0/1770573309 >> 172.21.15.64:0/1770573309 conn(0x7fd63c06daa0 msgr2=0x7fd63c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:20.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.176+0000 7fd641672640 1 -- 172.21.15.64:0/1770573309 shutdown_connections 2024-09-17T15:54:20.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.176+0000 7fd641672640 1 -- 172.21.15.64:0/1770573309 wait complete. 2024-09-17T15:54:20.188 INFO:tasks.ceph:Scrubbing osd.5 2024-09-17T15:54:20.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-17T15:54:20.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.291+0000 7f8814c47640 10 monclient: get_monmap_and_config 2024-09-17T15:54:20.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.291+0000 7f8814c47640 10 monclient: build_initial_monmap 2024-09-17T15:54:20.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.291+0000 7f8814c47640 10 monclient: monmap: 2024-09-17T15:54:20.293 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:20.292693+0000 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:20.292693+0000 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:20.294 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:20.295 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.292+0000 7f8814c47640 1 Processor -- start 2024-09-17T15:54:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.292+0000 7f8814c47640 1 -- start start 2024-09-17T15:54:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.292+0000 7f8814c47640 10 monclient: init 2024-09-17T15:54:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f8814c47640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f8814c47640 10 monclient: _add_conns ranks=[1,8,0,3,7,6,4,5,2] 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f8814c47640 10 monclient(hunting): picked mon.noname-f con 0x7f8810138d80 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f8814c47640 10 monclient(hunting): picked mon.noname-e con 0x7f88100a5de0 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f8814c47640 10 monclient(hunting): picked mon.noname-a con 0x7f881009cd20 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f8814c47640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88100a8390 con 0x7f881009cd20 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f8814c47640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f881009db80 con 0x7f8810138d80 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f8814c47640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f881009de60 con 0x7f88100a5de0 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f8814c47640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f8814c47640 10 monclient(hunting): authenticate will time out at 1257.968170s 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880dd9b640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f88100a5de0 0x7f881013e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51134/0 (socket says 172.21.15.64:51134) 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880dd9b640 1 -- 172.21.15.64:0/766924898 learned_addr learned my addr 172.21.15.64:0/766924898 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880ed9d640 1 --1- 172.21.15.64:0/766924898 >> v1:172.21.15.64:6789/0 conn(0x7f881009cd20 0x7f8810141830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35358/0 (socket says 172.21.15.64:35358) 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 1 -- 172.21.15.64:0/766924898 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 833526416 0 0) 0x7f881009de60 con 0x7f88100a5de0 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient(hunting): my global_id is 174275 2024-09-17T15:54:20.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 1 -- 172.21.15.64:0/766924898 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f8003520 con 0x7f88100a5de0 2024-09-17T15:54:20.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 1 -- 172.21.15.64:0/766924898 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1481090600 0 0) 0x7f88100a8390 con 0x7f881009cd20 2024-09-17T15:54:20.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient(hunting): my global_id is 125217 2024-09-17T15:54:20.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 1 -- 172.21.15.64:0/766924898 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f881009de60 con 0x7f881009cd20 2024-09-17T15:54:20.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 1 -- 172.21.15.64:0/766924898 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2685421624 0 0) 0x7f881009de60 con 0x7f881009cd20 2024-09-17T15:54:20.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 1 -- 172.21.15.64:0/766924898 >> v1:172.21.15.64:6793/0 conn(0x7f88100a5de0 legacy=0x7f881013e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 1 -- 172.21.15.64:0/766924898 >> v1:172.21.15.100:6789/0 conn(0x7f8810138d80 legacy=0x7f88100a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient: _finish_hunting 0 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 1 monclient: found mon.noname-a 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 1 -- 172.21.15.64:0/766924898 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881009e140 con 0x7f881009cd20 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient: _finish_auth 0 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 1 -- 172.21.15.64:0/766924898 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8808001ac0 con 0x7f881009cd20 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f8814c47640 5 monclient: authenticate success, global_id 125217 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f8814c47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f880d59a640 10 monclient: dump: 2024-09-17T15:54:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.305 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.293+0000 7f8814c47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f880d59a640 1 -- 172.21.15.64:0/766924898 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8808001ce0 con 0x7f881009cd20 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f880d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f880d59a640 1 -- 172.21.15.64:0/766924898 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8808004ea0 con 0x7f881009cd20 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f8814c47640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f8814c47640 4 set_mon_vals no callback set 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f880d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f880d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f880d59a640 10 monclient: dump: 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.306 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f8814c47640 10 monclient: shutdown 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f8814c47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:20.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f8814c47640 1 -- 172.21.15.64:0/766924898 >> v1:172.21.15.64:6789/0 conn(0x7f881009cd20 legacy=0x7f8810141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f8814c47640 1 -- 172.21.15.64:0/766924898 shutdown_connections 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f8814c47640 1 -- 172.21.15.64:0/766924898 >> 172.21.15.64:0/766924898 conn(0x7f881006dc70 msgr2=0x7f88100abb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f8814c47640 1 -- 172.21.15.64:0/766924898 shutdown_connections 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.294+0000 7f8814c47640 1 -- 172.21.15.64:0/766924898 wait complete. 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.295+0000 7f8814c47640 10 monclient: build_initial_monmap 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.295+0000 7f8814c47640 10 monclient: monmap: 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:20.308 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:20.309 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:20.309 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:20.309 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:20.309 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:20.309 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:20.309 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:20.309 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:20.309 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.295+0000 7f8814c47640 1 Processor -- start 2024-09-17T15:54:20.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.295+0000 7f8814c47640 1 -- start start 2024-09-17T15:54:20.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.295+0000 7f8814c47640 10 monclient: init 2024-09-17T15:54:20.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:20.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 10 monclient: _add_conns ranks=[7,2,1,6,5,8,4,0,3] 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 10 monclient(hunting): picked mon.noname-i con 0x7f881009cd20 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 10 monclient(hunting): picked mon.noname-b con 0x7f88100a5de0 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 10 monclient(hunting): picked mon.noname-f con 0x7f8810138d80 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88101326b0 con 0x7f8810138d80 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88100a5a00 con 0x7f88100a5de0 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8810130350 con 0x7f881009cd20 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 10 monclient(hunting): authenticate will time out at 1257.971127s 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f880dd9b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f88100a5de0 0x7f88101d2030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53670/0 (socket says 172.21.15.64:53670) 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f880dd9b640 1 -- 172.21.15.64:0/3601979534 learned_addr learned my addr 172.21.15.64:0/3601979534 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1092040160 0 0) 0x7f88100a5a00 con 0x7f88100a5de0 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): my global_id is 125336 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f0003520 con 0x7f88100a5de0 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3276627962 0 0) 0x7f8810130350 con 0x7f881009cd20 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): my global_id is 174328 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88100a5a00 con 0x7f881009cd20 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2662614890 0 0) 0x7f88101326b0 con 0x7f8810138d80 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): my global_id is 7039 2024-09-17T15:54:20.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8810130350 con 0x7f8810138d80 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 47185256 0 0) 0x7f87f0003520 con 0x7f88100a5de0 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88101326b0 con 0x7f88100a5de0 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8804001650 con 0x7f88100a5de0 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): dump: 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 439593836 0 0) 0x7f88101326b0 con 0x7f88100a5de0 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 >> v1:172.21.15.100:6792/0 conn(0x7f881009cd20 legacy=0x7f8810132ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 >> v1:172.21.15.100:6789/0 conn(0x7f8810138d80 legacy=0x7f88101d5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient: _finish_hunting 0 2024-09-17T15:54:20.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 monclient: found mon.b 2024-09-17T15:54:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8810131e10 con 0x7f88100a5de0 2024-09-17T15:54:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient: _finish_auth 0 2024-09-17T15:54:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 10 monclient: _check_auth_tickets 2024-09-17T15:54:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 5 monclient: authenticate success, global_id 125336 2024-09-17T15:54:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 10 monclient: _renew_subs 2024-09-17T15:54:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 1 -- 172.21.15.64:0/3601979534 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8810130eb0 con 0x7f88100a5de0 2024-09-17T15:54:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 10 monclient: _renew_subs 2024-09-17T15:54:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f8814c47640 1 -- 172.21.15.64:0/3601979534 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f88101d6e80 con 0x7f88100a5de0 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.296+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8804004a90 con 0x7f88100a5de0 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.297+0000 7f87feffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.297+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f8804004f80 con 0x7f88100a5de0 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.297+0000 7f87feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.297+0000 7f87feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.297+0000 7f87feffd640 10 monclient: dump: 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.297+0000 7f880cd99640 4 set_mon_vals no callback set 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.297+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f88040118e0 con 0x7f88100a5de0 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.297+0000 7f8814c47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.64:6790/0 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.297+0000 7f8814c47640 1 -- 172.21.15.64:0/3601979534 --> v1:172.21.15.64:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8810142f50 con 0x7f88100a5de0 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.298+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f880404eb80 con 0x7f88100a5de0 2024-09-17T15:54:20.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.298+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 --> v1:172.21.15.100:6800/4145568194 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87f004adb0 con 0x7f87f0047140 2024-09-17T15:54:20.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.298+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_get_version_reply(handle=1 version=327) ==== 24+0+0 (unknown 900077626 0 0) 0x7f8804013630 con 0x7f88100a5de0 2024-09-17T15:54:20.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.298+0000 7f87feffd640 10 monclient: handle_get_version_reply finishing 1 version 327 2024-09-17T15:54:20.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.298+0000 7f880e59c640 10 monclient: get_auth_request con 0x7f87f00405d0 auth_method 0 2024-09-17T15:54:20.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.298+0000 7f880ed9d640 10 monclient: get_auth_request con 0x7f87f0047140 auth_method 0 2024-09-17T15:54:20.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.299+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 <== osd.5 v1:172.21.15.100:6800/4145568194 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f87f004adb0 con 0x7f87f0047140 2024-09-17T15:54:20.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.311+0000 7f8814c47640 1 -- 172.21.15.64:0/3601979534 --> v1:172.21.15.100:6800/4145568194 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f881012c770 con 0x7f87f0047140 2024-09-17T15:54:20.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.312+0000 7f87feffd640 1 -- 172.21.15.64:0/3601979534 <== osd.5 v1:172.21.15.100:6800/4145568194 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f881012c770 con 0x7f87f0047140 2024-09-17T15:54:20.318 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-17T15:54:20.318 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T15:54:20.318 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-17T15:54:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.312+0000 7f8814c47640 1 -- 172.21.15.64:0/3601979534 >> v1:172.21.15.100:6800/4145568194 conn(0x7f87f0047140 legacy=0x7f87f0049560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.312+0000 7f8814c47640 1 -- 172.21.15.64:0/3601979534 >> v1:172.21.15.100:6812/4210963484 conn(0x7f87f00405d0 legacy=0x7f87f0042a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.312+0000 7f8814c47640 10 monclient: shutdown 2024-09-17T15:54:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.312+0000 7f8814c47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.312+0000 7f8814c47640 1 -- 172.21.15.64:0/3601979534 >> v1:172.21.15.64:6790/0 conn(0x7f88100a5de0 legacy=0x7f88101d2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.312+0000 7f880ed9d640 1 -- 172.21.15.64:0/3601979534 reap_dead start 2024-09-17T15:54:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.312+0000 7f8814c47640 1 -- 172.21.15.64:0/3601979534 shutdown_connections 2024-09-17T15:54:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.312+0000 7f8814c47640 1 -- 172.21.15.64:0/3601979534 >> 172.21.15.64:0/3601979534 conn(0x7f881006dc70 msgr2=0x7f88100a6400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.312+0000 7f8814c47640 1 -- 172.21.15.64:0/3601979534 shutdown_connections 2024-09-17T15:54:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.312+0000 7f8814c47640 1 -- 172.21.15.64:0/3601979534 wait complete. 2024-09-17T15:54:20.324 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-17T15:54:20.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.424+0000 7f89ef8d4640 10 monclient: get_monmap_and_config 2024-09-17T15:54:20.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.424+0000 7f89ef8d4640 10 monclient: build_initial_monmap 2024-09-17T15:54:20.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.424+0000 7f89ef8d4640 10 monclient: monmap: 2024-09-17T15:54:20.426 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:20.426 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.426 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:20.425494+0000 2024-09-17T15:54:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:20.425494+0000 2024-09-17T15:54:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:20.427 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.425+0000 7f89ef8d4640 1 Processor -- start 2024-09-17T15:54:20.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.425+0000 7f89ef8d4640 1 -- start start 2024-09-17T15:54:20.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.425+0000 7f89ef8d4640 10 monclient: init 2024-09-17T15:54:20.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.425+0000 7f89ef8d4640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:20.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.425+0000 7f89ef8d4640 10 monclient: _add_conns ranks=[2,1,5,4,7,0,3,8,6] 2024-09-17T15:54:20.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.425+0000 7f89ef8d4640 10 monclient(hunting): picked mon.noname-b con 0x7f89e80a5520 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:20.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.425+0000 7f89ef8d4640 10 monclient(hunting): picked mon.noname-f con 0x7f89e8135b50 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:20.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.425+0000 7f89ef8d4640 10 monclient(hunting): picked mon.noname-h con 0x7f89e8135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:20.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.425+0000 7f89ef8d4640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e809d6f0 con 0x7f89e8135b50 2024-09-17T15:54:20.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.425+0000 7f89ef8d4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e809d9d0 con 0x7f89e80a5520 2024-09-17T15:54:20.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.425+0000 7f89ef8d4640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e809dcb0 con 0x7f89e8135f70 2024-09-17T15:54:20.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.425+0000 7f89ef8d4640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:20.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.425+0000 7f89ef8d4640 10 monclient(hunting): authenticate will time out at 1258.100946s 2024-09-17T15:54:20.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89ed670640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f89e80a5520 0x7f89e80a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53678/0 (socket says 172.21.15.64:53678) 2024-09-17T15:54:20.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89ed670640 1 -- 172.21.15.64:0/3854298682 learned_addr learned my addr 172.21.15.64:0/3854298682 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:20.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 -- 172.21.15.64:0/3854298682 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1961847281 0 0) 0x7f89e809d9d0 con 0x7f89e80a5520 2024-09-17T15:54:20.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient(hunting): my global_id is 125345 2024-09-17T15:54:20.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 -- 172.21.15.64:0/3854298682 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89d4003520 con 0x7f89e80a5520 2024-09-17T15:54:20.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 -- 172.21.15.64:0/3854298682 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2119044926 0 0) 0x7f89e809dcb0 con 0x7f89e8135f70 2024-09-17T15:54:20.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient(hunting): my global_id is 174353 2024-09-17T15:54:20.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 -- 172.21.15.64:0/3854298682 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89e809d9d0 con 0x7f89e8135f70 2024-09-17T15:54:20.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 -- 172.21.15.64:0/3854298682 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3559195718 0 0) 0x7f89e809d6f0 con 0x7f89e8135b50 2024-09-17T15:54:20.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient(hunting): my global_id is 7048 2024-09-17T15:54:20.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 -- 172.21.15.64:0/3854298682 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89e809dcb0 con 0x7f89e8135b50 2024-09-17T15:54:20.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 -- 172.21.15.64:0/3854298682 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3545583497 0 0) 0x7f89d4003520 con 0x7f89e80a5520 2024-09-17T15:54:20.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 -- 172.21.15.64:0/3854298682 >> v1:172.21.15.100:6791/0 conn(0x7f89e8135f70 legacy=0x7f89e81415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 -- 172.21.15.64:0/3854298682 >> v1:172.21.15.100:6789/0 conn(0x7f89e8135b50 legacy=0x7f89e813deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient: _finish_hunting 0 2024-09-17T15:54:20.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 monclient: found mon.noname-b 2024-09-17T15:54:20.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 -- 172.21.15.64:0/3854298682 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e809ccd0 con 0x7f89e80a5520 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient: _finish_auth 0 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient: _check_auth_tickets 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89ef8d4640 5 monclient: authenticate success, global_id 125345 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89ef8d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 -- 172.21.15.64:0/3854298682 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f89e40016f0 con 0x7f89e80a5520 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient: dump: 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.441 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89ef8d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:20.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 -- 172.21.15.64:0/3854298682 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89e4001a00 con 0x7f89e80a5520 2024-09-17T15:54:20.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:20.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89ef8d4640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:20.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89ef8d4640 4 set_mon_vals no callback set 2024-09-17T15:54:20.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 1 -- 172.21.15.64:0/3854298682 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f89e4004be0 con 0x7f89e80a5520 2024-09-17T15:54:20.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:20.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:20.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.426+0000 7f89dffff640 10 monclient: dump: 2024-09-17T15:54:20.443 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.443 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.443 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.443 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.443 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.443 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.427+0000 7f89ef8d4640 10 monclient: shutdown 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.427+0000 7f89ef8d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.427+0000 7f89ef8d4640 1 -- 172.21.15.64:0/3854298682 >> v1:172.21.15.64:6790/0 conn(0x7f89e80a5520 legacy=0x7f89e80a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.427+0000 7f89ef8d4640 1 -- 172.21.15.64:0/3854298682 shutdown_connections 2024-09-17T15:54:20.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.427+0000 7f89ef8d4640 1 -- 172.21.15.64:0/3854298682 >> 172.21.15.64:0/3854298682 conn(0x7f89e806db30 msgr2=0x7f89e80ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:20.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.427+0000 7f89ef8d4640 1 -- 172.21.15.64:0/3854298682 shutdown_connections 2024-09-17T15:54:20.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.427+0000 7f89ef8d4640 1 -- 172.21.15.64:0/3854298682 wait complete. 2024-09-17T15:54:20.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.427+0000 7f89ef8d4640 10 monclient: build_initial_monmap 2024-09-17T15:54:20.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.427+0000 7f89ef8d4640 10 monclient: monmap: 2024-09-17T15:54:20.445 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:20.445 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:20.445 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:20.445 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:20.445 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:20.445 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:20.445 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:20.446 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:20.446 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:20.446 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:20.446 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:20.446 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:20.447 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:20.447 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:20.447 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:20.447 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ef8d4640 1 Processor -- start 2024-09-17T15:54:20.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ef8d4640 1 -- start start 2024-09-17T15:54:20.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ef8d4640 10 monclient: init 2024-09-17T15:54:20.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ef8d4640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:20.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ef8d4640 10 monclient: _add_conns ranks=[6,0,8,4,1,2,3,7,5] 2024-09-17T15:54:20.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ef8d4640 10 monclient(hunting): picked mon.noname-d con 0x7f89e80a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:20.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ef8d4640 10 monclient(hunting): picked mon.noname-a con 0x7f89e8135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:20.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ef8d4640 10 monclient(hunting): picked mon.noname-e con 0x7f89e8135f70 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:20.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ef8d4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e8142250 con 0x7f89e8135b50 2024-09-17T15:54:20.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ef8d4640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e813af50 con 0x7f89e80a5520 2024-09-17T15:54:20.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ef8d4640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e813b230 con 0x7f89e8135f70 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ef8d4640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ef8d4640 10 monclient(hunting): authenticate will time out at 1258.103860s 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ede71640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f89e8135f70 0x7f89e81d6a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:51144/0 (socket says 172.21.15.64:51144) 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ed670640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7f89e80a5520 0x7f89e81ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:54894/0 (socket says 172.21.15.64:54894) 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ede71640 1 -- 172.21.15.64:0/2537651251 learned_addr learned my addr 172.21.15.64:0/2537651251 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.428+0000 7f89ece6f640 1 --1- 172.21.15.64:0/2537651251 >> v1:172.21.15.64:6789/0 conn(0x7f89e8135b50 0x7f89e81d3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35366/0 (socket says 172.21.15.64:35366) 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3200197599 0 0) 0x7f89e813b230 con 0x7f89e8135f70 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): my global_id is 174284 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89cc003520 con 0x7f89e8135f70 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2280608393 0 0) 0x7f89e8142250 con 0x7f89e8135b50 2024-09-17T15:54:20.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): my global_id is 125226 2024-09-17T15:54:20.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89e813b230 con 0x7f89e8135b50 2024-09-17T15:54:20.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3019657085 0 0) 0x7f89e813af50 con 0x7f89e80a5520 2024-09-17T15:54:20.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): my global_id is 174282 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89e8142250 con 0x7f89e80a5520 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3053997287 0 0) 0x7f89e8142250 con 0x7f89e80a5520 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 --> v1:172.21.15.64:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89e813af50 con 0x7f89e80a5520 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3892067034 0 0) 0x7f89cc003520 con 0x7f89e8135f70 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89e8142250 con 0x7f89e8135f70 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f89e4004b20 con 0x7f89e80a5520 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): dump: 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2310477589 0 0) 0x7f89e813b230 con 0x7f89e8135b50 2024-09-17T15:54:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89cc003520 con 0x7f89e8135b50 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f89e0001e30 con 0x7f89e8135f70 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): dump: 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f89d8001a00 con 0x7f89e8135b50 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient(hunting): dump: 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.455 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.455 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.455 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.455 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.455 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.455 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.455 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.455 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.455 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.455 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.8 v1:172.21.15.64:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4280815447 0 0) 0x7f89e8142250 con 0x7f89e8135f70 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 >> v1:172.21.15.64:6792/0 conn(0x7f89e80a5520 legacy=0x7f89e81ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 >> v1:172.21.15.64:6789/0 conn(0x7f89e8135b50 legacy=0x7f89e81d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient: _finish_hunting 0 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 monclient: found mon.e 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 --> v1:172.21.15.64:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e8139a30 con 0x7f89e8135f70 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient: _finish_auth 0 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient: _check_auth_tickets 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:20.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ef8d4640 5 monclient: authenticate success, global_id 174284 2024-09-17T15:54:20.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ef8d4640 10 monclient: _renew_subs 2024-09-17T15:54:20.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ef8d4640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:20.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ef8d4640 1 -- 172.21.15.64:0/2537651251 --> v1:172.21.15.64:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f89e8070500 con 0x7f89e8135f70 2024-09-17T15:54:20.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.8 v1:172.21.15.64:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89e0003c40 con 0x7f89e8135f70 2024-09-17T15:54:20.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:20.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.8 v1:172.21.15.64:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f89e0004170 con 0x7f89e8135f70 2024-09-17T15:54:20.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:20.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:20.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89ddffb640 10 monclient: dump: 2024-09-17T15:54:20.457 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.457 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.458 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.458 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.458 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.458 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.458 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.458 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.458 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.459 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.459 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.459 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.459 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.459 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.459 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.459 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.429+0000 7f89df7fe640 4 set_mon_vals no callback set 2024-09-17T15:54:20.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.430+0000 7f89ef8d4640 10 monclient: _renew_subs 2024-09-17T15:54:20.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.430+0000 7f89ef8d4640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:20.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.430+0000 7f89ef8d4640 1 -- 172.21.15.64:0/2537651251 --> v1:172.21.15.64:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f89e813a6d0 con 0x7f89e8135f70 2024-09-17T15:54:20.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.430+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.8 v1:172.21.15.64:6793/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f89e0004390 con 0x7f89e8135f70 2024-09-17T15:54:20.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.431+0000 7f89ef8d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:20.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.431+0000 7f89ef8d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:20.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.431+0000 7f89ef8d4640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.64:6793/0 2024-09-17T15:54:20.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.431+0000 7f89ef8d4640 1 -- 172.21.15.64:0/2537651251 --> v1:172.21.15.64:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89e8138ed0 con 0x7f89e8135f70 2024-09-17T15:54:20.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.431+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.8 v1:172.21.15.64:6793/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f89e004eda0 con 0x7f89e8135f70 2024-09-17T15:54:20.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.431+0000 7f89ed670640 10 monclient: get_auth_request con 0x7f89cc040c90 auth_method 0 2024-09-17T15:54:20.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.435+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mon.8 v1:172.21.15.64:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f89e00179f0 con 0x7f89e8135f70 2024-09-17T15:54:20.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.435+0000 7f89ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:20.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.435+0000 7f89ddffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:20.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.624+0000 7f89ef8d4640 1 -- 172.21.15.64:0/2537651251 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f89e8134d90 con 0x7f89cc040c90 2024-09-17T15:54:20.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.625+0000 7f89ddffb640 1 -- 172.21.15.64:0/2537651251 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (unknown 2264484800 0 0) 0x7f89e8134d90 con 0x7f89cc040c90 2024-09-17T15:54:20.627 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 5 to deep-scrub 2024-09-17T15:54:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.627+0000 7f89ef8d4640 1 -- 172.21.15.64:0/2537651251 >> v1:172.21.15.100:6812/4210963484 conn(0x7f89cc040c90 legacy=0x7f89cc043150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.627+0000 7f89ef8d4640 10 monclient: shutdown 2024-09-17T15:54:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.627+0000 7f89ef8d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.627+0000 7f89ef8d4640 1 -- 172.21.15.64:0/2537651251 >> v1:172.21.15.64:6793/0 conn(0x7f89e8135f70 legacy=0x7f89e81d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.627+0000 7f89ef8d4640 1 -- 172.21.15.64:0/2537651251 shutdown_connections 2024-09-17T15:54:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.627+0000 7f89ef8d4640 1 -- 172.21.15.64:0/2537651251 >> 172.21.15.64:0/2537651251 conn(0x7f89e806db30 msgr2=0x7f89e80a89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.627+0000 7f89ef8d4640 1 -- 172.21.15.64:0/2537651251 shutdown_connections 2024-09-17T15:54:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.627+0000 7f89ef8d4640 1 -- 172.21.15.64:0/2537651251 wait complete. 2024-09-17T15:54:20.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:54:20.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.740+0000 7fb247364640 10 monclient: get_monmap_and_config 2024-09-17T15:54:20.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.740+0000 7fb247364640 10 monclient: build_initial_monmap 2024-09-17T15:54:20.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.740+0000 7fb247364640 10 monclient: monmap: 2024-09-17T15:54:20.742 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:20.742 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.742 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:20.741251+0000 2024-09-17T15:54:20.742 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:20.741251+0000 2024-09-17T15:54:20.742 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:20.743 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.743 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:20.743 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:20.743 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:20.743 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:20.743 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:20.743 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:20.743 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:20.743 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:20.743 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:20.743 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb247364640 1 Processor -- start 2024-09-17T15:54:20.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb247364640 1 -- start start 2024-09-17T15:54:20.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb247364640 10 monclient: init 2024-09-17T15:54:20.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb247364640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:20.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb247364640 10 monclient: _add_conns ranks=[4,0,6,1,2,7,5,8,3] 2024-09-17T15:54:20.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb247364640 10 monclient(hunting): picked mon.noname-c con 0x7fb2400a5520 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:20.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb247364640 10 monclient(hunting): picked mon.noname-a con 0x7fb240135b50 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:20.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb247364640 10 monclient(hunting): picked mon.noname-d con 0x7fb240135f70 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:20.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb247364640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb24009d6f0 con 0x7fb240135b50 2024-09-17T15:54:20.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb247364640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb24009d9d0 con 0x7fb2400a5520 2024-09-17T15:54:20.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb247364640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb24009dcb0 con 0x7fb240135f70 2024-09-17T15:54:20.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb247364640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:20.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb247364640 10 monclient(hunting): authenticate will time out at 1258.416939s 2024-09-17T15:54:20.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb245901640 1 --1- >> v1:172.21.15.64:6792/0 conn(0x7fb240135f70 0x7fb2401415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:54900/0 (socket says 172.21.15.64:54900) 2024-09-17T15:54:20.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb245100640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fb2400a5520 0x7fb2400a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:59916/0 (socket says 172.21.15.64:59916) 2024-09-17T15:54:20.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.741+0000 7fb245901640 1 -- 172.21.15.64:0/1528822968 learned_addr learned my addr 172.21.15.64:0/1528822968 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:20.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 -- 172.21.15.64:0/1528822968 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 965254563 0 0) 0x7fb24009d9d0 con 0x7fb2400a5520 2024-09-17T15:54:20.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient(hunting): my global_id is 154786 2024-09-17T15:54:20.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 -- 172.21.15.64:0/1528822968 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb22c003520 con 0x7fb2400a5520 2024-09-17T15:54:20.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 -- 172.21.15.64:0/1528822968 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1774253934 0 0) 0x7fb24009dcb0 con 0x7fb240135f70 2024-09-17T15:54:20.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient(hunting): my global_id is 174291 2024-09-17T15:54:20.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 -- 172.21.15.64:0/1528822968 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb24009d9d0 con 0x7fb240135f70 2024-09-17T15:54:20.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 -- 172.21.15.64:0/1528822968 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 374600728 0 0) 0x7fb24009d6f0 con 0x7fb240135b50 2024-09-17T15:54:20.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient(hunting): my global_id is 125235 2024-09-17T15:54:20.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 -- 172.21.15.64:0/1528822968 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb24009dcb0 con 0x7fb240135b50 2024-09-17T15:54:20.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 -- 172.21.15.64:0/1528822968 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 348480701 0 0) 0x7fb22c003520 con 0x7fb2400a5520 2024-09-17T15:54:20.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 -- 172.21.15.64:0/1528822968 >> v1:172.21.15.64:6792/0 conn(0x7fb240135f70 legacy=0x7fb2401415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 -- 172.21.15.64:0/1528822968 >> v1:172.21.15.64:6789/0 conn(0x7fb240135b50 legacy=0x7fb24013deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient: _finish_hunting 0 2024-09-17T15:54:20.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 monclient: found mon.noname-c 2024-09-17T15:54:20.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:20.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.64:6791/0 2024-09-17T15:54:20.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 -- 172.21.15.64:0/1528822968 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24009ccd0 con 0x7fb2400a5520 2024-09-17T15:54:20.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient: _finish_auth 0 2024-09-17T15:54:20.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient: _check_auth_tickets 2024-09-17T15:54:20.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:20.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 -- 172.21.15.64:0/1528822968 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb23c001830 con 0x7fb2400a5520 2024-09-17T15:54:20.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:20.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:20.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient: dump: 2024-09-17T15:54:20.761 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.761 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.761 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.761 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.762 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.762 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.762 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.762 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.762 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.762 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.762 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.762 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.762 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.762 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.763 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.763 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb247364640 5 monclient: authenticate success, global_id 154786 2024-09-17T15:54:20.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb247364640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:20.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 -- 172.21.15.64:0/1528822968 <== mon.4 v1:172.21.15.64:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb23c001a50 con 0x7fb2400a5520 2024-09-17T15:54:20.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:20.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb247364640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:20.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 1 -- 172.21.15.64:0/1528822968 <== mon.4 v1:172.21.15.64:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb23c001d40 con 0x7fb2400a5520 2024-09-17T15:54:20.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:20.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb247364640 4 set_mon_vals no callback set 2024-09-17T15:54:20.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:20.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.742+0000 7fb237fff640 10 monclient: dump: 2024-09-17T15:54:20.764 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.764 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.764 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.764 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.764 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.764 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.765 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.765 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.765 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.765 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.765 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.766 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.766 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.766 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.766 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.766 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.743+0000 7fb247364640 10 monclient: shutdown 2024-09-17T15:54:20.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.743+0000 7fb247364640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:20.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.743+0000 7fb247364640 1 -- 172.21.15.64:0/1528822968 >> v1:172.21.15.64:6791/0 conn(0x7fb2400a5520 legacy=0x7fb2400a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.743+0000 7fb247364640 1 -- 172.21.15.64:0/1528822968 shutdown_connections 2024-09-17T15:54:20.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.743+0000 7fb247364640 1 -- 172.21.15.64:0/1528822968 >> 172.21.15.64:0/1528822968 conn(0x7fb24006db30 msgr2=0x7fb2400ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:20.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.743+0000 7fb247364640 1 -- 172.21.15.64:0/1528822968 shutdown_connections 2024-09-17T15:54:20.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.743+0000 7fb247364640 1 -- 172.21.15.64:0/1528822968 wait complete. 2024-09-17T15:54:20.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.743+0000 7fb247364640 10 monclient: build_initial_monmap 2024-09-17T15:54:20.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.743+0000 7fb247364640 10 monclient: monmap: 2024-09-17T15:54:20.768 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:20.768 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:20.768 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:20.768 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:20.769 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:20.769 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:20.769 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:20.769 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:20.769 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:20.769 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:20.769 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:20.769 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:20.769 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:20.770 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:20.771 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:20.771 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb247364640 1 Processor -- start 2024-09-17T15:54:20.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb247364640 1 -- start start 2024-09-17T15:54:20.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb247364640 10 monclient: init 2024-09-17T15:54:20.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb247364640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:20.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb247364640 10 monclient: _add_conns ranks=[7,2,0,6,3,1,8,4,5] 2024-09-17T15:54:20.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb247364640 10 monclient(hunting): picked mon.noname-i con 0x7fb2400a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:20.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb247364640 10 monclient(hunting): picked mon.noname-b con 0x7fb240135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:20.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb247364640 10 monclient(hunting): picked mon.noname-a con 0x7fb240135f70 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:20.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb247364640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2400a0780 con 0x7fb240135f70 2024-09-17T15:54:20.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb247364640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2400a2880 con 0x7fb240135b50 2024-09-17T15:54:20.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb247364640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2400a92a0 con 0x7fb2400a5520 2024-09-17T15:54:20.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb247364640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:20.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb247364640 10 monclient(hunting): authenticate will time out at 1258.419927s 2024-09-17T15:54:20.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.744+0000 7fb245901640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fb240135f70 0x7fb2401d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35386/0 (socket says 172.21.15.64:35386) 2024-09-17T15:54:20.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb245901640 1 -- 172.21.15.64:0/1677560635 learned_addr learned my addr 172.21.15.64:0/1677560635 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:20.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb2448ff640 1 --1- 172.21.15.64:0/1677560635 >> v1:172.21.15.64:6790/0 conn(0x7fb240135b50 0x7fb24009f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53684/0 (socket says 172.21.15.64:53684) 2024-09-17T15:54:20.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1943173578 0 0) 0x7fb2400a0780 con 0x7fb240135f70 2024-09-17T15:54:20.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): my global_id is 125244 2024-09-17T15:54:20.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb224003520 con 0x7fb240135f70 2024-09-17T15:54:20.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4230636713 0 0) 0x7fb2400a2880 con 0x7fb240135b50 2024-09-17T15:54:20.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): my global_id is 125354 2024-09-17T15:54:20.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2400a0780 con 0x7fb240135b50 2024-09-17T15:54:20.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1047634915 0 0) 0x7fb2400a92a0 con 0x7fb2400a5520 2024-09-17T15:54:20.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:20.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:20.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:20.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): my global_id is 174337 2024-09-17T15:54:20.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2400a2880 con 0x7fb2400a5520 2024-09-17T15:54:20.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1731625546 0 0) 0x7fb224003520 con 0x7fb240135f70 2024-09-17T15:54:20.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2400a92a0 con 0x7fb240135f70 2024-09-17T15:54:20.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb238001df0 con 0x7fb240135f70 2024-09-17T15:54:20.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:20.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:54:20.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): dump: 2024-09-17T15:54:20.774 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.774 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.774 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.774 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.774 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.774 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.774 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.774 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.774 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.775 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.775 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.775 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.775 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.775 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.775 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.775 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:20.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1785994693 0 0) 0x7fb2400a0780 con 0x7fb240135b50 2024-09-17T15:54:20.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb224003520 con 0x7fb240135b50 2024-09-17T15:54:20.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb230001290 con 0x7fb240135b50 2024-09-17T15:54:20.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:20.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-17T15:54:20.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient(hunting): dump: 2024-09-17T15:54:20.776 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.776 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.776 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.776 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.777 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.777 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.777 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.777 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.777 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.777 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.777 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.777 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.777 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2577608656 0 0) 0x7fb2400a92a0 con 0x7fb240135f70 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 >> v1:172.21.15.100:6792/0 conn(0x7fb2400a5520 legacy=0x7fb24009ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 >> v1:172.21.15.64:6790/0 conn(0x7fb240135b50 legacy=0x7fb24009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 monclient: found mon.a 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2400a8d70 con 0x7fb240135f70 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient: _finish_auth 0 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:54:20.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:20.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb247364640 5 monclient: authenticate success, global_id 125244 2024-09-17T15:54:20.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb247364640 10 monclient: _renew_subs 2024-09-17T15:54:20.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb247364640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:20.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb247364640 1 -- 172.21.15.64:0/1677560635 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb240135040 con 0x7fb240135f70 2024-09-17T15:54:20.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb238004e10 con 0x7fb240135f70 2024-09-17T15:54:20.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:20.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fb238005300 con 0x7fb240135f70 2024-09-17T15:54:20.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:20.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:20.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb235ffb640 10 monclient: dump: 2024-09-17T15:54:20.779 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:20.779 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:20.779 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:20.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.745+0000 7fb2377fe640 4 set_mon_vals no callback set 2024-09-17T15:54:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.746+0000 7fb247364640 10 monclient: _renew_subs 2024-09-17T15:54:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.746+0000 7fb247364640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.746+0000 7fb247364640 1 -- 172.21.15.64:0/1677560635 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb2400a22c0 con 0x7fb240135f70 2024-09-17T15:54:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.746+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fb238011be0 con 0x7fb240135f70 2024-09-17T15:54:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.747+0000 7fb247364640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.747+0000 7fb247364640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.747+0000 7fb247364640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-09-17T15:54:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.747+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7fb23804edb0 con 0x7fb240135f70 2024-09-17T15:54:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.747+0000 7fb247364640 1 -- 172.21.15.64:0/1677560635 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2400a7890 con 0x7fb240135f70 2024-09-17T15:54:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.747+0000 7fb245100640 10 monclient: get_auth_request con 0x7fb2240408a0 auth_method 0 2024-09-17T15:54:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.751+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fb2400a7890 con 0x7fb240135f70 2024-09-17T15:54:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.751+0000 7fb235ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:20.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.751+0000 7fb235ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:20.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.931+0000 7fb247364640 1 -- 172.21.15.64:0/1677560635 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb2400a2660 con 0x7fb2240408a0 2024-09-17T15:54:20.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.933+0000 7fb235ffb640 1 -- 172.21.15.64:0/1677560635 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34003 (unknown 2965378022 0 4143621120) 0x7fb2400a2660 con 0x7fb2240408a0 2024-09-17T15:54:20.935 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:54:20.935 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-17T15:54:20.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.935+0000 7fb247364640 1 -- 172.21.15.64:0/1677560635 >> v1:172.21.15.100:6812/4210963484 conn(0x7fb2240408a0 legacy=0x7fb224042d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.935+0000 7fb247364640 10 monclient: shutdown 2024-09-17T15:54:20.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.935+0000 7fb247364640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:20.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.935+0000 7fb247364640 1 -- 172.21.15.64:0/1677560635 >> v1:172.21.15.64:6789/0 conn(0x7fb240135f70 legacy=0x7fb2401d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:20.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.935+0000 7fb247364640 1 -- 172.21.15.64:0/1677560635 shutdown_connections 2024-09-17T15:54:20.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.935+0000 7fb247364640 1 -- 172.21.15.64:0/1677560635 >> 172.21.15.64:0/1677560635 conn(0x7fb24006db30 msgr2=0x7fb2400a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:20.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.935+0000 7fb247364640 1 -- 172.21.15.64:0/1677560635 shutdown_connections 2024-09-17T15:54:20.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:20.935+0000 7fb247364640 1 -- 172.21.15.64:0/1677560635 wait complete. 2024-09-17T15:54:20.948 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":575,"stamp":"2024-09-17T15:54:19.940821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":183872,"kb_used_data":4900,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566047168,"statfs":{"total":579820584960,"available":579632300032,"internally_reserved":0,"allocated":5017600,"data_stored":2520766,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":9568,"internal_metadata":182966944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002741"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":635,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.592448+0000","last_change":"2024-09-17T15:46:03.953098+0000","last_active":"2024-09-17T15:53:06.592448+0000","last_peered":"2024-09-17T15:53:06.592448+0000","last_clean":"2024-09-17T15:53:06.592448+0000","last_became_active":"2024-09-17T15:46:02.372737+0000","last_became_peered":"2024-09-17T15:46:02.372737+0000","last_unstale":"2024-09-17T15:53:06.592448+0000","last_undegraded":"2024-09-17T15:53:06.592448+0000","last_fullsized":"2024-09-17T15:53:06.592448+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:04:26.679936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057658000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":627,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.594073+0000","last_change":"2024-09-17T15:46:03.954609+0000","last_active":"2024-09-17T15:53:06.594073+0000","last_peered":"2024-09-17T15:53:06.594073+0000","last_clean":"2024-09-17T15:53:06.594073+0000","last_became_active":"2024-09-17T15:46:01.943967+0000","last_became_peered":"2024-09-17T15:46:01.943967+0000","last_unstale":"2024-09-17T15:53:06.594073+0000","last_undegraded":"2024-09-17T15:53:06.594073+0000","last_fullsized":"2024-09-17T15:53:06.594073+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:44:26.433116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058926900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":635,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.592297+0000","last_change":"2024-09-17T15:46:03.953301+0000","last_active":"2024-09-17T15:53:06.592297+0000","last_peered":"2024-09-17T15:53:06.592297+0000","last_clean":"2024-09-17T15:53:06.592297+0000","last_became_active":"2024-09-17T15:46:01.937233+0000","last_became_peered":"2024-09-17T15:46:01.937233+0000","last_unstale":"2024-09-17T15:53:06.592297+0000","last_undegraded":"2024-09-17T15:53:06.592297+0000","last_fullsized":"2024-09-17T15:53:06.592297+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:36:15.542838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001175981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":627,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.592454+0000","last_change":"2024-09-17T15:46:03.954824+0000","last_active":"2024-09-17T15:53:06.592454+0000","last_peered":"2024-09-17T15:53:06.592454+0000","last_clean":"2024-09-17T15:53:06.592454+0000","last_became_active":"2024-09-17T15:46:01.941419+0000","last_became_peered":"2024-09-17T15:46:01.941419+0000","last_unstale":"2024-09-17T15:53:06.592454+0000","last_undegraded":"2024-09-17T15:53:06.592454+0000","last_fullsized":"2024-09-17T15:53:06.592454+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:53:39.894895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077909099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":633,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.593738+0000","last_change":"2024-09-17T15:46:03.952714+0000","last_active":"2024-09-17T15:53:06.593738+0000","last_peered":"2024-09-17T15:53:06.593738+0000","last_clean":"2024-09-17T15:53:06.593738+0000","last_became_active":"2024-09-17T15:46:01.941164+0000","last_became_peered":"2024-09-17T15:46:01.941164+0000","last_unstale":"2024-09-17T15:53:06.593738+0000","last_undegraded":"2024-09-17T15:53:06.593738+0000","last_fullsized":"2024-09-17T15:53:06.593738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:07:59.957722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051738399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":629,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.709842+0000","last_change":"2024-09-17T15:46:03.954913+0000","last_active":"2024-09-17T15:53:06.709842+0000","last_peered":"2024-09-17T15:53:06.709842+0000","last_clean":"2024-09-17T15:53:06.709842+0000","last_became_active":"2024-09-17T15:46:01.941188+0000","last_became_peered":"2024-09-17T15:46:01.941188+0000","last_unstale":"2024-09-17T15:53:06.709842+0000","last_undegraded":"2024-09-17T15:53:06.709842+0000","last_fullsized":"2024-09-17T15:53:06.709842+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:25:32.291552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087819899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":636,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.591629+0000","last_change":"2024-09-17T15:46:03.953722+0000","last_active":"2024-09-17T15:53:06.591629+0000","last_peered":"2024-09-17T15:53:06.591629+0000","last_clean":"2024-09-17T15:53:06.591629+0000","last_became_active":"2024-09-17T15:46:01.941223+0000","last_became_peered":"2024-09-17T15:46:01.941223+0000","last_unstale":"2024-09-17T15:53:06.591629+0000","last_undegraded":"2024-09-17T15:53:06.591629+0000","last_fullsized":"2024-09-17T15:53:06.591629+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:18:15.696629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012017600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":632,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.594399+0000","last_change":"2024-09-17T15:46:03.952788+0000","last_active":"2024-09-17T15:53:06.594399+0000","last_peered":"2024-09-17T15:53:06.594399+0000","last_clean":"2024-09-17T15:53:06.594399+0000","last_became_active":"2024-09-17T15:46:01.937420+0000","last_became_peered":"2024-09-17T15:46:01.937420+0000","last_unstale":"2024-09-17T15:53:06.594399+0000","last_undegraded":"2024-09-17T15:53:06.594399+0000","last_fullsized":"2024-09-17T15:53:06.594399+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:46:00.918933+0000","last_clean_scrub_stamp":"2024-09-17T15:46:00.918933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:03:18.015826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065159300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'72","reported_seq":737,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:53:06.591745+0000","last_change":"2024-09-17T15:45:56.919389+0000","last_active":"2024-09-17T15:53:06.591745+0000","last_peered":"2024-09-17T15:53:06.591745+0000","last_clean":"2024-09-17T15:53:06.591745+0000","last_became_active":"2024-09-17T15:45:56.908629+0000","last_became_peered":"2024-09-17T15:45:56.908629+0000","last_unstale":"2024-09-17T15:53:06.591745+0000","last_undegraded":"2024-09-17T15:53:06.591745+0000","last_fullsized":"2024-09-17T15:53:06.591745+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:45:53.875287+0000","last_clean_scrub_stamp":"2024-09-17T15:45:53.875287+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:44:47.696204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":1057856,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":14,"seq":60129542250,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":819200,"data_stored":243812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542249,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":823296,"data_stored":243831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542251,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":860160,"data_stored":772740,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":819200,"data_stored":243812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44480,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327360,"statfs":{"total":96636764160,"available":96591216640,"internally_reserved":0,"allocated":823296,"data_stored":243831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640361,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":872448,"data_stored":772740,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:54:20.952 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-17T15:46:00.918933+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=46, tm_sec=0, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=54, tm_sec=16, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:54:20.952 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-17T15:46:00.918933+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=46, tm_sec=0, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=54, tm_sec=16, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:54:20.952 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-17T15:46:00.918933+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=46, tm_sec=0, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=54, tm_sec=16, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:54:20.952 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-17T15:46:00.918933+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=46, tm_sec=0, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=54, tm_sec=16, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:54:20.953 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-17T15:46:00.918933+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=46, tm_sec=0, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=54, tm_sec=16, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:54:20.953 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-17T15:46:00.918933+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=46, tm_sec=0, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=54, tm_sec=16, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:54:20.953 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-17T15:46:00.918933+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=46, tm_sec=0, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=54, tm_sec=16, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:54:20.953 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-17T15:46:00.918933+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=46, tm_sec=0, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=54, tm_sec=16, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:54:20.953 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-17T15:45:53.875287+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=45, tm_sec=53, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=54, tm_sec=16, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:54:20.953 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-17T15:54:40.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:54:41.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.067+0000 7fcad92b0640 10 monclient: get_monmap_and_config 2024-09-17T15:54:41.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.067+0000 7fcad92b0640 10 monclient: build_initial_monmap 2024-09-17T15:54:41.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.067+0000 7fcad92b0640 10 monclient: monmap: 2024-09-17T15:54:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:41.068224+0000 2024-09-17T15:54:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:41.068224+0000 2024-09-17T15:54:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:41.071 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.067+0000 7fcad92b0640 1 Processor -- start 2024-09-17T15:54:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad92b0640 1 -- start start 2024-09-17T15:54:41.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad92b0640 10 monclient: init 2024-09-17T15:54:41.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad92b0640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:41.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad92b0640 10 monclient: _add_conns ranks=[6,2,5,7,1,4,3,8,0] 2024-09-17T15:54:41.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad92b0640 10 monclient(hunting): picked mon.noname-d con 0x7fcad40a5520 addr v1:172.21.15.64:6792/0 2024-09-17T15:54:41.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad92b0640 10 monclient(hunting): picked mon.noname-b con 0x7fcad4135b50 addr v1:172.21.15.64:6790/0 2024-09-17T15:54:41.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad92b0640 10 monclient(hunting): picked mon.noname-h con 0x7fcad4135f70 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:41.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad92b0640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad409d6f0 con 0x7fcad4135b50 2024-09-17T15:54:41.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad92b0640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad409d9d0 con 0x7fcad4135f70 2024-09-17T15:54:41.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad92b0640 1 -- --> v1:172.21.15.64:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad409dcb0 con 0x7fcad40a5520 2024-09-17T15:54:41.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad92b0640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:41.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad92b0640 10 monclient(hunting): authenticate will time out at 1278.743589s 2024-09-17T15:54:41.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad27fc640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fcad4135b50 0x7fcad413deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:43260/0 (socket says 172.21.15.64:43260) 2024-09-17T15:54:41.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad27fc640 1 -- 172.21.15.64:0/829781240 learned_addr learned my addr 172.21.15.64:0/829781240 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:41.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad2ffd640 1 --1- 172.21.15.64:0/829781240 >> v1:172.21.15.64:6792/0 conn(0x7fcad40a5520 0x7fcad40a5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6792/0 says I am v1:172.21.15.64:54808/0 (socket says 172.21.15.64:54808) 2024-09-17T15:54:41.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad1ffb640 1 -- 172.21.15.64:0/829781240 <== mon.6 v1:172.21.15.64:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422663597 0 0) 0x7fcad409dcb0 con 0x7fcad40a5520 2024-09-17T15:54:41.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad1ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:41.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:41.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:41.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.068+0000 7fcad1ffb640 10 monclient(hunting): my global_id is 174300 2024-09-17T15:54:41.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 1 -- 172.21.15.64:0/829781240 --> v1:172.21.15.64:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcabc003520 con 0x7fcad40a5520 2024-09-17T15:54:41.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 1 -- 172.21.15.64:0/829781240 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1917227396 0 0) 0x7fcad409d6f0 con 0x7fcad4135b50 2024-09-17T15:54:41.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:41.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:41.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:41.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient(hunting): my global_id is 125363 2024-09-17T15:54:41.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 1 -- 172.21.15.64:0/829781240 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcad409dcb0 con 0x7fcad4135b50 2024-09-17T15:54:41.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 1 -- 172.21.15.64:0/829781240 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2349785264 0 0) 0x7fcad409d9d0 con 0x7fcad4135f70 2024-09-17T15:54:41.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:41.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:41.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:41.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient(hunting): my global_id is 174362 2024-09-17T15:54:41.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 1 -- 172.21.15.64:0/829781240 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcad409d6f0 con 0x7fcad4135f70 2024-09-17T15:54:41.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 1 -- 172.21.15.64:0/829781240 <== mon.6 v1:172.21.15.64:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 287558378 0 0) 0x7fcabc003520 con 0x7fcad40a5520 2024-09-17T15:54:41.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 1 -- 172.21.15.64:0/829781240 >> v1:172.21.15.100:6791/0 conn(0x7fcad4135f70 legacy=0x7fcad41415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 1 -- 172.21.15.64:0/829781240 >> v1:172.21.15.64:6790/0 conn(0x7fcad4135b50 legacy=0x7fcad413deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:54:41.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 1 monclient: found mon.noname-d 2024-09-17T15:54:41.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:41.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.64:6792/0 2024-09-17T15:54:41.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 1 -- 172.21.15.64:0/829781240 --> v1:172.21.15.64:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad409ccd0 con 0x7fcad40a5520 2024-09-17T15:54:41.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:54:41.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:54:41.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:41.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad92b0640 5 monclient: authenticate success, global_id 174300 2024-09-17T15:54:41.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad92b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:41.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 1 -- 172.21.15.64:0/829781240 <== mon.6 v1:172.21.15.64:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcac4001830 con 0x7fcad40a5520 2024-09-17T15:54:41.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:41.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-17T15:54:41.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient: dump: 2024-09-17T15:54:41.089 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:41.089 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.089 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.089 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.089 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:41.089 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.089 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:41.090 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:41.090 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:41.090 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:41.090 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:41.090 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:41.090 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:41.090 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:41.090 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:41.090 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad92b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:41.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 1 -- 172.21.15.64:0/829781240 <== mon.6 v1:172.21.15.64:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcac4001a50 con 0x7fcad40a5520 2024-09-17T15:54:41.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:41.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad92b0640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 1 -- 172.21.15.64:0/829781240 <== mon.6 v1:172.21.15.64:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcac4001d40 con 0x7fcad40a5520 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad92b0640 4 set_mon_vals no callback set 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad1ffb640 10 monclient: dump: 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:41.091 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:41.092 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:41.092 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:41.092 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:41.092 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:41.092 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:41.092 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad92b0640 10 monclient: shutdown 2024-09-17T15:54:41.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad92b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:41.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.069+0000 7fcad92b0640 1 -- 172.21.15.64:0/829781240 >> v1:172.21.15.64:6792/0 conn(0x7fcad40a5520 legacy=0x7fcad40a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.070+0000 7fcad92b0640 1 -- 172.21.15.64:0/829781240 shutdown_connections 2024-09-17T15:54:41.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.070+0000 7fcad92b0640 1 -- 172.21.15.64:0/829781240 >> 172.21.15.64:0/829781240 conn(0x7fcad406db30 msgr2=0x7fcad40ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:41.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.070+0000 7fcad92b0640 1 -- 172.21.15.64:0/829781240 shutdown_connections 2024-09-17T15:54:41.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.070+0000 7fcad92b0640 1 -- 172.21.15.64:0/829781240 wait complete. 2024-09-17T15:54:41.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.070+0000 7fcad92b0640 10 monclient: build_initial_monmap 2024-09-17T15:54:41.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.070+0000 7fcad92b0640 10 monclient: monmap: 2024-09-17T15:54:41.093 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:41.093 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:41.093 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:41.093 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:41.094 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:41.094 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:41.094 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:41.094 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:41.094 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:41.094 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:41.094 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:41.094 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:41.094 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:41.094 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:41.094 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad92b0640 1 Processor -- start 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad92b0640 1 -- start start 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad92b0640 10 monclient: init 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad92b0640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad92b0640 10 monclient: _add_conns ranks=[7,5,4,1,6,8,3,0,2] 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad92b0640 10 monclient(hunting): picked mon.noname-i con 0x7fcad40a5520 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad92b0640 10 monclient(hunting): picked mon.noname-h con 0x7fcad4135b50 addr v1:172.21.15.100:6791/0 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad92b0640 10 monclient(hunting): picked mon.noname-c con 0x7fcad4135f70 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad92b0640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad40a9150 con 0x7fcad4135f70 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad92b0640 1 -- --> v1:172.21.15.100:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad40a0600 con 0x7fcad4135b50 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad92b0640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad4134ee0 con 0x7fcad40a5520 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad92b0640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:41.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad92b0640 10 monclient(hunting): authenticate will time out at 1278.746764s 2024-09-17T15:54:41.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad37fe640 1 --1- >> v1:172.21.15.64:6791/0 conn(0x7fcad4135f70 0x7fcad41d6200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6791/0 says I am v1:172.21.15.64:42384/0 (socket says 172.21.15.64:42384) 2024-09-17T15:54:41.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.071+0000 7fcad37fe640 1 -- 172.21.15.64:0/1338139342 learned_addr learned my addr 172.21.15.64:0/1338139342 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:41.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 182269 0 0) 0x7fcad409d6f0 con 0x7fcad4135f70 2024-09-17T15:54:41.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:41.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:41.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:41.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): my global_id is 154795 2024-09-17T15:54:41.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcac0003550 con 0x7fcad4135f70 2024-09-17T15:54:41.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.5 v1:172.21.15.100:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2842312012 0 0) 0x7fcad40a0600 con 0x7fcad4135b50 2024-09-17T15:54:41.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:41.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:41.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:41.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): my global_id is 174371 2024-09-17T15:54:41.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 --> v1:172.21.15.100:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcad409d6f0 con 0x7fcad4135b50 2024-09-17T15:54:41.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161441725 0 0) 0x7fcad4134ee0 con 0x7fcad40a5520 2024-09-17T15:54:41.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:54:41.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:41.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:41.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): my global_id is 174346 2024-09-17T15:54:41.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcad40a0600 con 0x7fcad40a5520 2024-09-17T15:54:41.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2093181262 0 0) 0x7fcac0003550 con 0x7fcad4135f70 2024-09-17T15:54:41.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcad4134ee0 con 0x7fcad4135f70 2024-09-17T15:54:41.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcacc001e10 con 0x7fcad4135f70 2024-09-17T15:54:41.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:41.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:41.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): dump: 2024-09-17T15:54:41.099 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:41.099 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.099 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.099 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.099 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:41.099 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.099 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:41.099 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:41.099 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:41.099 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:41.100 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:41.100 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:41.100 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:41.100 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:41.100 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:41.100 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:41.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.5 v1:172.21.15.100:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1239777790 0 0) 0x7fcad409d6f0 con 0x7fcad4135b50 2024-09-17T15:54:41.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 --> v1:172.21.15.100:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcac0003550 con 0x7fcad4135b50 2024-09-17T15:54:41.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.5 v1:172.21.15.100:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcac8001430 con 0x7fcad4135b50 2024-09-17T15:54:41.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:41.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-17T15:54:41.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): dump: 2024-09-17T15:54:41.101 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:41.101 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.101 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.101 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.101 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:41.101 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.102 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:41.102 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:41.102 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:41.102 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:41.102 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:41.102 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:41.102 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:41.102 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:41.102 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:41.103 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1021540234 0 0) 0x7fcad40a0600 con 0x7fcad40a5520 2024-09-17T15:54:41.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcad409d6f0 con 0x7fcad40a5520 2024-09-17T15:54:41.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcac40017f0 con 0x7fcad40a5520 2024-09-17T15:54:41.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:41.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:54:41.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient(hunting): dump: 2024-09-17T15:54:41.104 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:41.104 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.104 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.104 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.104 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:41.105 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.105 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:41.105 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:41.105 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:41.105 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:41.105 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:41.105 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:41.106 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:41.106 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:41.106 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:41.106 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2488562973 0 0) 0x7fcad4134ee0 con 0x7fcad4135f70 2024-09-17T15:54:41.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 >> v1:172.21.15.100:6792/0 conn(0x7fcad40a5520 legacy=0x7fcad40a4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 >> v1:172.21.15.100:6791/0 conn(0x7fcad4135b50 legacy=0x7fcad41d2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:54:41.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 monclient: found mon.c 2024-09-17T15:54:41.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:41.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:41.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad40a0320 con 0x7fcad4135f70 2024-09-17T15:54:41.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:54:41.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:54:41.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:41.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcad92b0640 5 monclient: authenticate success, global_id 154795 2024-09-17T15:54:41.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcad92b0640 10 monclient: _renew_subs 2024-09-17T15:54:41.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcad92b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:41.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcad92b0640 1 -- 172.21.15.64:0/1338139342 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcad40a5250 con 0x7fcad4135f70 2024-09-17T15:54:41.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcad92b0640 10 monclient: _renew_subs 2024-09-17T15:54:41.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcad92b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:41.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcad92b0640 1 -- 172.21.15.64:0/1338139342 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fcad41cd750 con 0x7fcad4135f70 2024-09-17T15:54:41.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcacc003c40 con 0x7fcad4135f70 2024-09-17T15:54:41.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:41.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7fcacc004270 con 0x7fcad4135f70 2024-09-17T15:54:41.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:41.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcad17fa640 4 set_mon_vals no callback set 2024-09-17T15:54:41.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:41.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.072+0000 7fcabb7fe640 10 monclient: dump: 2024-09-17T15:54:41.109 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:41.109 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.109 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.109 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.110 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:41.110 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.110 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:41.110 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:41.110 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:41.110 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:41.110 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:41.110 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:41.110 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:41.111 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:41.111 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:41.111 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.073+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7fcacc004490 con 0x7fcad4135f70 2024-09-17T15:54:41.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.074+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7fcacc04ed50 con 0x7fcad4135f70 2024-09-17T15:54:41.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.074+0000 7fcad92b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:41.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.074+0000 7fcad92b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:41.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.074+0000 7fcad92b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:41.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.074+0000 7fcad92b0640 1 -- 172.21.15.64:0/1338139342 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcab4005080 con 0x7fcad4135f70 2024-09-17T15:54:41.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.074+0000 7fcad2ffd640 10 monclient: get_auth_request con 0x7fcac0040cc0 auth_method 0 2024-09-17T15:54:41.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.079+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7fcad40a9150 con 0x7fcad4135f70 2024-09-17T15:54:41.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.079+0000 7fcabb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:41.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.079+0000 7fcabb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:41.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.262+0000 7fcad92b0640 1 -- 172.21.15.64:0/1338139342 --> v1:172.21.15.100:6812/4210963484 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcab4000b90 con 0x7fcac0040cc0 2024-09-17T15:54:41.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.263+0000 7fcabb7fe640 1 -- 172.21.15.64:0/1338139342 <== mgr.4109 v1:172.21.15.100:6812/4210963484 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34020 (unknown 2965378022 0 3158508208) 0x7fcab4000b90 con 0x7fcac0040cc0 2024-09-17T15:54:41.266 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-17T15:54:41.266 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-17T15:54:41.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.266+0000 7fcad92b0640 1 -- 172.21.15.64:0/1338139342 >> v1:172.21.15.100:6812/4210963484 conn(0x7fcac0040cc0 legacy=0x7fcac0043180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.266+0000 7fcad92b0640 10 monclient: shutdown 2024-09-17T15:54:41.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.266+0000 7fcad92b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:41.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.266+0000 7fcad92b0640 1 -- 172.21.15.64:0/1338139342 >> v1:172.21.15.64:6791/0 conn(0x7fcad4135f70 legacy=0x7fcad41d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.266+0000 7fcad92b0640 1 -- 172.21.15.64:0/1338139342 shutdown_connections 2024-09-17T15:54:41.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.266+0000 7fcad92b0640 1 -- 172.21.15.64:0/1338139342 >> 172.21.15.64:0/1338139342 conn(0x7fcad406db30 msgr2=0x7fcad412c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:41.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.266+0000 7fcad92b0640 1 -- 172.21.15.64:0/1338139342 shutdown_connections 2024-09-17T15:54:41.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.266+0000 7fcad92b0640 1 -- 172.21.15.64:0/1338139342 wait complete. 2024-09-17T15:54:41.279 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":585,"stamp":"2024-09-17T15:54:39.944791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":183788,"kb_used_data":4900,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566047252,"statfs":{"total":579820584960,"available":579632386048,"internally_reserved":0,"allocated":5017600,"data_stored":2520766,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":9568,"internal_metadata":182966944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002390"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":643,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:54:20.704325+0000","last_change":"2024-09-17T15:54:20.704325+0000","last_active":"2024-09-17T15:54:20.704325+0000","last_peered":"2024-09-17T15:54:20.704325+0000","last_clean":"2024-09-17T15:54:20.704325+0000","last_became_active":"2024-09-17T15:46:02.372737+0000","last_became_peered":"2024-09-17T15:46:02.372737+0000","last_unstale":"2024-09-17T15:54:20.704325+0000","last_undegraded":"2024-09-17T15:54:20.704325+0000","last_fullsized":"2024-09-17T15:54:20.704325+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:54:20.704244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:54:20.704244+0000","last_clean_scrub_stamp":"2024-09-17T15:54:20.704244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:45:40.976433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057658000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":635,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:54:19.965177+0000","last_change":"2024-09-17T15:54:19.965177+0000","last_active":"2024-09-17T15:54:19.965177+0000","last_peered":"2024-09-17T15:54:19.965177+0000","last_clean":"2024-09-17T15:54:19.965177+0000","last_became_active":"2024-09-17T15:46:01.943967+0000","last_became_peered":"2024-09-17T15:46:01.943967+0000","last_unstale":"2024-09-17T15:54:19.965177+0000","last_undegraded":"2024-09-17T15:54:19.965177+0000","last_fullsized":"2024-09-17T15:54:19.965177+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:54:19.965115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:54:19.965115+0000","last_clean_scrub_stamp":"2024-09-17T15:54:19.965115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:19:59.930006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058926900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":644,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:54:21.655821+0000","last_change":"2024-09-17T15:54:21.655728+0000","last_active":"2024-09-17T15:54:21.655821+0000","last_peered":"2024-09-17T15:54:21.655821+0000","last_clean":"2024-09-17T15:54:21.655821+0000","last_became_active":"2024-09-17T15:46:01.937233+0000","last_became_peered":"2024-09-17T15:46:01.937233+0000","last_unstale":"2024-09-17T15:54:21.655821+0000","last_undegraded":"2024-09-17T15:54:21.655821+0000","last_fullsized":"2024-09-17T15:54:21.655821+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:54:21.655635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:54:21.655635+0000","last_clean_scrub_stamp":"2024-09-17T15:54:21.655635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:56:44.605359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001175981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":636,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:54:18.917847+0000","last_change":"2024-09-17T15:54:18.917762+0000","last_active":"2024-09-17T15:54:18.917847+0000","last_peered":"2024-09-17T15:54:18.917847+0000","last_clean":"2024-09-17T15:54:18.917847+0000","last_became_active":"2024-09-17T15:46:01.941419+0000","last_became_peered":"2024-09-17T15:46:01.941419+0000","last_unstale":"2024-09-17T15:54:18.917847+0000","last_undegraded":"2024-09-17T15:54:18.917847+0000","last_fullsized":"2024-09-17T15:54:18.917847+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:54:18.917691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:54:18.917691+0000","last_clean_scrub_stamp":"2024-09-17T15:54:18.917691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:37:22.850845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077909099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"327'3","reported_seq":643,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:54:21.965120+0000","last_change":"2024-09-17T15:54:21.965064+0000","last_active":"2024-09-17T15:54:21.965120+0000","last_peered":"2024-09-17T15:54:21.965120+0000","last_clean":"2024-09-17T15:54:21.965120+0000","last_became_active":"2024-09-17T15:46:01.941164+0000","last_became_peered":"2024-09-17T15:46:01.941164+0000","last_unstale":"2024-09-17T15:54:21.965120+0000","last_undegraded":"2024-09-17T15:54:21.965120+0000","last_fullsized":"2024-09-17T15:54:21.965120+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'3","last_scrub_stamp":"2024-09-17T15:54:21.964985+0000","last_deep_scrub":"327'3","last_deep_scrub_stamp":"2024-09-17T15:54:21.964985+0000","last_clean_scrub_stamp":"2024-09-17T15:54:21.964985+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:18:43.200502+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051738399999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":638,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:54:19.367343+0000","last_change":"2024-09-17T15:54:19.367297+0000","last_active":"2024-09-17T15:54:19.367343+0000","last_peered":"2024-09-17T15:54:19.367343+0000","last_clean":"2024-09-17T15:54:19.367343+0000","last_became_active":"2024-09-17T15:46:01.941188+0000","last_became_peered":"2024-09-17T15:46:01.941188+0000","last_unstale":"2024-09-17T15:54:19.367343+0000","last_undegraded":"2024-09-17T15:54:19.367343+0000","last_fullsized":"2024-09-17T15:54:19.367343+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:54:19.367235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:54:19.367235+0000","last_clean_scrub_stamp":"2024-09-17T15:54:19.367235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:22:09.396624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087819899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":644,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:54:22.626717+0000","last_change":"2024-09-17T15:54:22.626717+0000","last_active":"2024-09-17T15:54:22.626717+0000","last_peered":"2024-09-17T15:54:22.626717+0000","last_clean":"2024-09-17T15:54:22.626717+0000","last_became_active":"2024-09-17T15:46:01.941223+0000","last_became_peered":"2024-09-17T15:46:01.941223+0000","last_unstale":"2024-09-17T15:54:22.626717+0000","last_undegraded":"2024-09-17T15:54:22.626717+0000","last_fullsized":"2024-09-17T15:54:22.626717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:54:22.626636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:54:22.626636+0000","last_clean_scrub_stamp":"2024-09-17T15:54:22.626636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:54:30.808766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012017600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"327'2","reported_seq":642,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:54:20.997238+0000","last_change":"2024-09-17T15:54:20.997139+0000","last_active":"2024-09-17T15:54:20.997238+0000","last_peered":"2024-09-17T15:54:20.997238+0000","last_clean":"2024-09-17T15:54:20.997238+0000","last_became_active":"2024-09-17T15:46:01.937420+0000","last_became_peered":"2024-09-17T15:46:01.937420+0000","last_unstale":"2024-09-17T15:54:20.997238+0000","last_undegraded":"2024-09-17T15:54:20.997238+0000","last_fullsized":"2024-09-17T15:54:20.997238+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'2","last_scrub_stamp":"2024-09-17T15:54:20.997048+0000","last_deep_scrub":"327'2","last_deep_scrub_stamp":"2024-09-17T15:54:20.997048+0000","last_clean_scrub_stamp":"2024-09-17T15:54:20.997048+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:25:49.644329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065159300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"327'74","reported_seq":747,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-17T15:54:19.717762+0000","last_change":"2024-09-17T15:54:19.717762+0000","last_active":"2024-09-17T15:54:19.717762+0000","last_peered":"2024-09-17T15:54:19.717762+0000","last_clean":"2024-09-17T15:54:19.717762+0000","last_became_active":"2024-09-17T15:45:56.908629+0000","last_became_peered":"2024-09-17T15:45:56.908629+0000","last_unstale":"2024-09-17T15:54:19.717762+0000","last_undegraded":"2024-09-17T15:54:19.717762+0000","last_fullsized":"2024-09-17T15:54:19.717762+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'74","last_scrub_stamp":"2024-09-17T15:54:19.717666+0000","last_deep_scrub":"327'74","last_deep_scrub_stamp":"2024-09-17T15:54:19.717666+0000","last_clean_scrub_stamp":"2024-09-17T15:54:19.717666+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:48:51.602883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":1057856,"data_compressed":11848,"data_compressed_allocated":65536,"data_compressed_original":1036288,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":14,"seq":60129542254,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":819200,"data_stored":243812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542253,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":243831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542255,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":860160,"data_stored":772740,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":819200,"data_stored":243812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640365,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44468,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327372,"statfs":{"total":96636764160,"available":96591228928,"internally_reserved":0,"allocated":823296,"data_stored":243831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640365,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":772740,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6333,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5515,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:54:41.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-17T15:54:41.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.379+0000 7f2db58fe640 10 monclient: get_monmap_and_config 2024-09-17T15:54:41.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.379+0000 7f2db58fe640 10 monclient: build_initial_monmap 2024-09-17T15:54:41.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.379+0000 7f2db58fe640 10 monclient: monmap: 2024-09-17T15:54:41.381 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:41.381 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.381 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:54:41.380431+0000 2024-09-17T15:54:41.382 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:54:41.380431+0000 2024-09-17T15:54:41.382 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:41.382 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.382 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:41.382 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:41.382 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:41.382 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:41.382 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:41.382 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:41.382 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:41.382 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:41.382 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:41.382 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2db58fe640 1 Processor -- start 2024-09-17T15:54:41.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2db58fe640 1 -- start start 2024-09-17T15:54:41.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2db58fe640 10 monclient: init 2024-09-17T15:54:41.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2db58fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:41.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2db58fe640 10 monclient: _add_conns ranks=[1,0,7,6,2,4,8,5,3] 2024-09-17T15:54:41.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2db58fe640 10 monclient(hunting): picked mon.noname-f con 0x7f2db00a5600 addr v1:172.21.15.100:6789/0 2024-09-17T15:54:41.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2db58fe640 10 monclient(hunting): picked mon.noname-a con 0x7f2db01397e0 addr v1:172.21.15.64:6789/0 2024-09-17T15:54:41.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2db58fe640 10 monclient(hunting): picked mon.noname-i con 0x7f2db013f800 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:41.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2db58fe640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db009dab0 con 0x7f2db01397e0 2024-09-17T15:54:41.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2db58fe640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db009dd90 con 0x7f2db00a5600 2024-09-17T15:54:41.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2db58fe640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db009e070 con 0x7f2db013f800 2024-09-17T15:54:41.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2db58fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:41.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2db58fe640 10 monclient(hunting): authenticate will time out at 1279.055881s 2024-09-17T15:54:41.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2dae7fc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f2db01397e0 0x7f2db013e0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:58538/0 (socket says 172.21.15.64:58538) 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.380+0000 7f2dae7fc640 1 -- 172.21.15.64:0/527630302 learned_addr learned my addr 172.21.15.64:0/527630302 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 -- 172.21.15.64:0/527630302 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2472780117 0 0) 0x7f2da4001110 con 0x7f2db01397e0 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient(hunting): my global_id is 125253 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 -- 172.21.15.64:0/527630302 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d98003520 con 0x7f2db01397e0 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 -- 172.21.15.64:0/527630302 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3419102254 0 0) 0x7f2db009e070 con 0x7f2db013f800 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient(hunting): my global_id is 174355 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 -- 172.21.15.64:0/527630302 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2da4001110 con 0x7f2db013f800 2024-09-17T15:54:41.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 -- 172.21.15.64:0/527630302 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 112613286 0 0) 0x7f2db009dd90 con 0x7f2db00a5600 2024-09-17T15:54:41.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient(hunting): _negotiate 2024-09-17T15:54:41.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:41.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:41.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient(hunting): my global_id is 7057 2024-09-17T15:54:41.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 -- 172.21.15.64:0/527630302 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2db009e070 con 0x7f2db00a5600 2024-09-17T15:54:41.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 -- 172.21.15.64:0/527630302 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2925945244 0 0) 0x7f2d98003520 con 0x7f2db01397e0 2024-09-17T15:54:41.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 -- 172.21.15.64:0/527630302 >> v1:172.21.15.100:6792/0 conn(0x7f2db013f800 legacy=0x7f2db0141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 -- 172.21.15.64:0/527630302 >> v1:172.21.15.100:6789/0 conn(0x7f2db00a5600 legacy=0x7f2db00a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient: _finish_hunting 0 2024-09-17T15:54:41.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 monclient: found mon.noname-a 2024-09-17T15:54:41.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:41.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-09-17T15:54:41.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 -- 172.21.15.64:0/527630302 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db009ca00 con 0x7f2db01397e0 2024-09-17T15:54:41.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient: _finish_auth 0 2024-09-17T15:54:41.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient: _check_auth_tickets 2024-09-17T15:54:41.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:41.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 -- 172.21.15.64:0/527630302 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2da4001a00 con 0x7f2db01397e0 2024-09-17T15:54:41.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:41.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:54:41.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient: dump: 2024-09-17T15:54:41.393 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:41.393 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.393 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.393 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.393 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:41.393 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2db58fe640 5 monclient: authenticate success, global_id 125253 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2db58fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 -- 172.21.15.64:0/527630302 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2da4001c20 con 0x7f2db01397e0 2024-09-17T15:54:41.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 1 -- 172.21.15.64:0/527630302 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2da4004be0 con 0x7f2db01397e0 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2db58fe640 10 monclient: get_monmap_and_config success 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2db58fe640 4 set_mon_vals no callback set 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.381+0000 7f2dadffb640 10 monclient: dump: 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:41.395 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:41.396 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:41.396 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:41.396 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:41.396 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:41.396 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:41.396 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:41.396 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:41.396 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.382+0000 7f2db58fe640 10 monclient: shutdown 2024-09-17T15:54:41.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.382+0000 7f2db58fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:41.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.382+0000 7f2db58fe640 1 -- 172.21.15.64:0/527630302 >> v1:172.21.15.64:6789/0 conn(0x7f2db01397e0 legacy=0x7f2db013e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.382+0000 7f2db58fe640 1 -- 172.21.15.64:0/527630302 shutdown_connections 2024-09-17T15:54:41.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.382+0000 7f2db58fe640 1 -- 172.21.15.64:0/527630302 >> 172.21.15.64:0/527630302 conn(0x7f2db006dd40 msgr2=0x7f2db00aba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:41.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.382+0000 7f2db58fe640 1 -- 172.21.15.64:0/527630302 shutdown_connections 2024-09-17T15:54:41.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.382+0000 7f2db58fe640 1 -- 172.21.15.64:0/527630302 wait complete. 2024-09-17T15:54:41.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.382+0000 7f2db58fe640 10 monclient: build_initial_monmap 2024-09-17T15:54:41.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.382+0000 7f2db58fe640 10 monclient: monmap: 2024-09-17T15:54:41.397 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-09-17T15:54:41.397 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:54:41.397 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-09-17T15:54:41.397 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-09-17T15:54:41.398 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-09-17T15:54:41.398 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-09-17T15:54:41.398 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-09-17T15:54:41.398 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.noname-f 2024-09-17T15:54:41.398 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-09-17T15:54:41.398 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.noname-g 2024-09-17T15:54:41.398 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.noname-c 2024-09-17T15:54:41.398 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.noname-h 2024-09-17T15:54:41.398 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.noname-d 2024-09-17T15:54:41.398 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.noname-i 2024-09-17T15:54:41.398 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.noname-e 2024-09-17T15:54:41.398 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2db58fe640 1 Processor -- start 2024-09-17T15:54:41.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2db58fe640 1 -- start start 2024-09-17T15:54:41.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2db58fe640 10 monclient: init 2024-09-17T15:54:41.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2db58fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:54:41.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2db58fe640 10 monclient: _add_conns ranks=[7,4,8,0,1,2,5,6,3] 2024-09-17T15:54:41.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2db58fe640 10 monclient(hunting): picked mon.noname-i con 0x7f2db00a5600 addr v1:172.21.15.100:6792/0 2024-09-17T15:54:41.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2db58fe640 10 monclient(hunting): picked mon.noname-c con 0x7f2db01397e0 addr v1:172.21.15.64:6791/0 2024-09-17T15:54:41.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2db58fe640 10 monclient(hunting): picked mon.noname-e con 0x7f2db013f800 addr v1:172.21.15.64:6793/0 2024-09-17T15:54:41.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2db58fe640 1 -- --> v1:172.21.15.64:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db013d720 con 0x7f2db01397e0 2024-09-17T15:54:41.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2db58fe640 1 -- --> v1:172.21.15.100:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db00aa8a0 con 0x7f2db00a5600 2024-09-17T15:54:41.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2db58fe640 1 -- --> v1:172.21.15.64:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db0138780 con 0x7f2db013f800 2024-09-17T15:54:41.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2db58fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:54:41.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2db58fe640 10 monclient(hunting): authenticate will time out at 1279.058835s 2024-09-17T15:54:41.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2daf7fe640 1 --1- >> v1:172.21.15.64:6793/0 conn(0x7f2db013f800 0x7f2db01d6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6793/0 says I am v1:172.21.15.64:44302/0 (socket says 172.21.15.64:44302) 2024-09-17T15:54:41.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.383+0000 7f2daf7fe640 1 -- 172.21.15.64:0/3359837111 learned_addr learned my addr 172.21.15.64:0/3359837111 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-09-17T15:54:41.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.8 v1:172.21.15.64:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3229042837 0 0) 0x7f2db0138780 con 0x7f2db013f800 2024-09-17T15:54:41.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:41.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:41.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:41.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): my global_id is 174293 2024-09-17T15:54:41.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 --> v1:172.21.15.64:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d9c003520 con 0x7f2db013f800 2024-09-17T15:54:41.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.4 v1:172.21.15.64:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1467969921 0 0) 0x7f2db013d720 con 0x7f2db01397e0 2024-09-17T15:54:41.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:41.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:41.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): my global_id is 154804 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 --> v1:172.21.15.64:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2db0138780 con 0x7f2db01397e0 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.7 v1:172.21.15.100:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1568529122 0 0) 0x7f2db009e070 con 0x7f2db00a5600 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): _negotiate 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): my global_id is 174364 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 --> v1:172.21.15.100:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2db013d720 con 0x7f2db00a5600 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.4 v1:172.21.15.64:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2516239985 0 0) 0x7f2db0138780 con 0x7f2db01397e0 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 --> v1:172.21.15.64:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2db009e070 con 0x7f2db01397e0 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.4 v1:172.21.15.64:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2da4004b20 con 0x7f2db01397e0 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:54:41.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): dump: 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:41.402 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:41.403 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:41.403 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:54:41.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.8 v1:172.21.15.64:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2518581342 0 0) 0x7f2d9c003520 con 0x7f2db013f800 2024-09-17T15:54:41.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 --> v1:172.21.15.64:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2db0138780 con 0x7f2db013f800 2024-09-17T15:54:41.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.7 v1:172.21.15.100:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3434793265 0 0) 0x7f2db013d720 con 0x7f2db00a5600 2024-09-17T15:54:41.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 --> v1:172.21.15.100:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d9c003520 con 0x7f2db00a5600 2024-09-17T15:54:41.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.8 v1:172.21.15.64:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2da8001d70 con 0x7f2db013f800 2024-09-17T15:54:41.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:41.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-17T15:54:41.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): dump: 2024-09-17T15:54:41.403 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:41.403 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.404 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.404 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.404 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:41.404 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.404 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:41.404 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:41.404 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:41.404 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:41.404 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:41.404 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:41.404 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:41.404 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:41.404 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:41.405 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.7 v1:172.21.15.100:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2da00019d0 con 0x7f2db00a5600 2024-09-17T15:54:41.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-17T15:54:41.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-17T15:54:41.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient(hunting): dump: 2024-09-17T15:54:41.405 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:41.405 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.405 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.405 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.405 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:41.405 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.405 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.4 v1:172.21.15.64:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3489125135 0 0) 0x7f2db009e070 con 0x7f2db01397e0 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 >> v1:172.21.15.64:6793/0 conn(0x7f2db013f800 legacy=0x7f2db01d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 >> v1:172.21.15.100:6792/0 conn(0x7f2db00a5600 legacy=0x7f2db00a4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient: _finish_hunting 0 2024-09-17T15:54:41.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 monclient: found mon.c 2024-09-17T15:54:41.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:54:41.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:41.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 --> v1:172.21.15.64:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db009f110 con 0x7f2db01397e0 2024-09-17T15:54:41.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient: _finish_auth 0 2024-09-17T15:54:41.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient: _check_auth_tickets 2024-09-17T15:54:41.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:54:41.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2db58fe640 5 monclient: authenticate success, global_id 154804 2024-09-17T15:54:41.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2db58fe640 10 monclient: _renew_subs 2024-09-17T15:54:41.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2db58fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:41.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2db58fe640 1 -- 172.21.15.64:0/3359837111 --> v1:172.21.15.64:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2db009e570 con 0x7f2db01397e0 2024-09-17T15:54:41.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2db58fe640 10 monclient: _renew_subs 2024-09-17T15:54:41.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2db58fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:41.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2db58fe640 1 -- 172.21.15.64:0/3359837111 --> v1:172.21.15.64:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f2db00a4d90 con 0x7f2db01397e0 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.4 v1:172.21.15.64:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2da40012e0 con 0x7f2db01397e0 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.4 v1:172.21.15.64:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 3232502772 0 0) 0x7f2da4001aa0 con 0x7f2db01397e0 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2dad7fa640 4 set_mon_vals no callback set 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.384+0000 7f2d97fff640 10 monclient: dump: 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 3de99658-3189-432a-9efe-6c9d4c0a2934 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-09-17T15:45:15.994818+0000 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-09-17T15:54:41.408 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-09-17T15:54:41.409 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.100:6789/0 mon.f 2024-09-17T15:54:41.409 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.b 2024-09-17T15:54:41.409 INFO:teuthology.orchestra.run.smithi064.stderr:3: v1:172.21.15.100:6790/0 mon.g 2024-09-17T15:54:41.409 INFO:teuthology.orchestra.run.smithi064.stderr:4: v1:172.21.15.64:6791/0 mon.c 2024-09-17T15:54:41.409 INFO:teuthology.orchestra.run.smithi064.stderr:5: v1:172.21.15.100:6791/0 mon.h 2024-09-17T15:54:41.409 INFO:teuthology.orchestra.run.smithi064.stderr:6: v1:172.21.15.64:6792/0 mon.d 2024-09-17T15:54:41.409 INFO:teuthology.orchestra.run.smithi064.stderr:7: v1:172.21.15.100:6792/0 mon.i 2024-09-17T15:54:41.409 INFO:teuthology.orchestra.run.smithi064.stderr:8: v1:172.21.15.64:6793/0 mon.e 2024-09-17T15:54:41.409 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:41.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.385+0000 7f2db58fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:41.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.385+0000 7f2db58fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:41.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.385+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.4 v1:172.21.15.64:6791/0 7 ==== mgrmap(e 5) ==== 50996+0+0 (unknown 3654117632 0 0) 0x7f2da4011c10 con 0x7f2db01397e0 2024-09-17T15:54:41.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.385+0000 7f2db58fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:41.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.385+0000 7f2db58fe640 1 -- 172.21.15.64:0/3359837111 --> v1:172.21.15.64:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d90005080 con 0x7f2db01397e0 2024-09-17T15:54:41.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.385+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.4 v1:172.21.15.64:6791/0 8 ==== osd_map(327..327 src has 288..327) ==== 3506+0+0 (unknown 1716466290 0 0) 0x7f2da404ed20 con 0x7f2db01397e0 2024-09-17T15:54:41.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.386+0000 7f2daeffd640 10 monclient: get_auth_request con 0x7f2d9c040d70 auth_method 0 2024-09-17T15:54:41.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.390+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.4 v1:172.21.15.64:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (unknown 1092875540 0 2276561608) 0x7f2db009dab0 con 0x7f2db01397e0 2024-09-17T15:54:41.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.390+0000 7f2d97fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:54:41.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.390+0000 7f2d97fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:54:41.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.582+0000 7f2db58fe640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-17T15:54:41.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.582+0000 7f2db58fe640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-17T15:54:41.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.582+0000 7f2db58fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6791/0 2024-09-17T15:54:41.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.582+0000 7f2db58fe640 1 -- 172.21.15.64:0/3359837111 --> v1:172.21.15.64:6791/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f2d90005370 con 0x7f2db01397e0 2024-09-17T15:54:41.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.588+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.4 v1:172.21.15.64:6791/0 10 ==== config(1 keys) ==== 35+0+0 (unknown 2555425225 0 0) 0x7f2da4016e50 con 0x7f2db01397e0 2024-09-17T15:54:41.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.588+0000 7f2d97fff640 10 monclient: handle_config config(1 keys) 2024-09-17T15:54:41.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.588+0000 7f2dad7fa640 4 set_mon_vals no callback set 2024-09-17T15:54:41.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.588+0000 7f2dad7fa640 10 set_mon_vals mon_health_to_clog = false 2024-09-17T15:54:41.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.588+0000 7f2d97fff640 1 -- 172.21.15.64:0/3359837111 <== mon.4 v1:172.21.15.64:6791/0 11 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (unknown 1453024670 0 0) 0x7f2da4017970 con 0x7f2db01397e0 2024-09-17T15:54:41.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.588+0000 7f2d97fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-17T15:54:41.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.588+0000 7f2d97fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:54:41.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.591+0000 7f2db58fe640 1 -- 172.21.15.64:0/3359837111 >> v1:172.21.15.100:6812/4210963484 conn(0x7f2d9c040d70 legacy=0x7f2d9c043230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.591+0000 7f2db58fe640 10 monclient: shutdown 2024-09-17T15:54:41.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.591+0000 7f2db58fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:54:41.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.591+0000 7f2db58fe640 1 -- 172.21.15.64:0/3359837111 >> v1:172.21.15.64:6791/0 conn(0x7f2db01397e0 legacy=0x7f2db01d2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:54:41.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.591+0000 7f2db58fe640 1 -- 172.21.15.64:0/3359837111 shutdown_connections 2024-09-17T15:54:41.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.591+0000 7f2db58fe640 1 -- 172.21.15.64:0/3359837111 >> 172.21.15.64:0/3359837111 conn(0x7f2db006dd40 msgr2=0x7f2db00a9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:54:41.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.591+0000 7f2db58fe640 1 -- 172.21.15.64:0/3359837111 shutdown_connections 2024-09-17T15:54:41.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-17T15:54:41.591+0000 7f2db58fe640 1 -- 172.21.15.64:0/3359837111 wait complete. 2024-09-17T15:54:41.604 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-17T15:54:41.604 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-17T15:54:41.604 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-17T15:54:41.604 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:41.637 INFO:tasks.ceph.osd.0:Stopped 2024-09-17T15:54:41.637 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-17T15:54:41.637 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:41.712 INFO:tasks.ceph.osd.1:Stopped 2024-09-17T15:54:41.712 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-17T15:54:41.712 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:41.781 INFO:tasks.ceph.osd.2:Stopped 2024-09-17T15:54:41.782 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-17T15:54:41.782 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:41.817 INFO:tasks.ceph.osd.3:Stopped 2024-09-17T15:54:41.817 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-17T15:54:41.817 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:41.900 INFO:tasks.ceph.osd.4:Stopped 2024-09-17T15:54:41.900 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-17T15:54:41.900 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:41.935 INFO:tasks.ceph.osd.5:Stopped 2024-09-17T15:54:41.936 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-17T15:54:41.936 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-17T15:54:41.936 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:42.004 INFO:tasks.ceph.mgr.x:Stopped 2024-09-17T15:54:42.004 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-17T15:54:42.004 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T15:54:42.004 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:42.016 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T15:54:42.016 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:54:42.016 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:42.068 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:54:42.068 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T15:54:42.068 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:42.119 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T15:54:42.119 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-09-17T15:54:42.119 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:42.170 INFO:tasks.ceph.mon.d:Stopped 2024-09-17T15:54:42.171 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-09-17T15:54:42.171 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:42.221 INFO:tasks.ceph.mon.e:Stopped 2024-09-17T15:54:42.221 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-09-17T15:54:42.221 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:42.236 INFO:tasks.ceph.mon.f:Stopped 2024-09-17T15:54:42.236 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-09-17T15:54:42.236 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:42.289 INFO:tasks.ceph.mon.g:Stopped 2024-09-17T15:54:42.290 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-09-17T15:54:42.290 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:42.342 INFO:tasks.ceph.mon.h:Stopped 2024-09-17T15:54:42.342 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-09-17T15:54:42.342 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:54:42.394 INFO:tasks.ceph.mon.i:Stopped 2024-09-17T15:54:42.394 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-17T15:54:42.394 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | head -n 1 2024-09-17T15:54:42.429 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-17T15:54:42.429 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-17T15:54:42.616 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-17T15:54:42.616 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-17T15:54:42.688 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-17T15:54:42.688 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-17T15:54:42.756 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T15:54:42.756 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-17T15:54:42.956 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T15:54:42.956 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-17T15:54:43.026 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T15:54:43.027 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-17T15:54:43.076 INFO:tasks.ceph:Archiving mon data... 2024-09-17T15:54:43.077 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/data/mon.a.tgz 2024-09-17T15:54:43.077 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-09-17T15:54:43.094 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.TfuLh6QFub 2024-09-17T15:54:43.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.TfuLh6QFub 2024-09-17T15:54:43.207 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.TfuLh6QFub 2024-09-17T15:54:43.291 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.TfuLh6QFub is 225KB 2024-09-17T15:54:43.387 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.TfuLh6QFub 2024-09-17T15:54:43.402 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/data/mon.b.tgz 2024-09-17T15:54:43.403 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-09-17T15:54:43.457 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.s1xtxz8C4x 2024-09-17T15:54:43.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.s1xtxz8C4x 2024-09-17T15:54:43.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.s1xtxz8C4x 2024-09-17T15:54:43.641 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.s1xtxz8C4x is 225KB 2024-09-17T15:54:43.734 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.s1xtxz8C4x 2024-09-17T15:54:43.750 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/data/mon.c.tgz 2024-09-17T15:54:43.750 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-09-17T15:54:43.805 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.Zqns70XnS8 2024-09-17T15:54:43.805 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Zqns70XnS8 2024-09-17T15:54:43.904 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.Zqns70XnS8 2024-09-17T15:54:43.989 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.Zqns70XnS8 is 222KB 2024-09-17T15:54:44.084 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.Zqns70XnS8 2024-09-17T15:54:44.099 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-d to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/data/mon.d.tgz 2024-09-17T15:54:44.099 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-09-17T15:54:44.154 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.yiqo7JK8lf 2024-09-17T15:54:44.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-d -- . > /tmp/tmp.yiqo7JK8lf 2024-09-17T15:54:44.251 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.yiqo7JK8lf 2024-09-17T15:54:44.334 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.yiqo7JK8lf is 220KB 2024-09-17T15:54:44.428 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.yiqo7JK8lf 2024-09-17T15:54:44.444 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-e to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/data/mon.e.tgz 2024-09-17T15:54:44.444 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-09-17T15:54:44.499 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.pXRYvzckpZ 2024-09-17T15:54:44.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-e -- . > /tmp/tmp.pXRYvzckpZ 2024-09-17T15:54:44.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.pXRYvzckpZ 2024-09-17T15:54:44.680 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.pXRYvzckpZ is 220KB 2024-09-17T15:54:44.775 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.pXRYvzckpZ 2024-09-17T15:54:44.790 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/mon/ceph-f to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/data/mon.f.tgz 2024-09-17T15:54:44.791 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-09-17T15:54:44.809 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.wG3z2ACvc3 2024-09-17T15:54:44.810 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-f -- . > /tmp/tmp.wG3z2ACvc3 2024-09-17T15:54:44.927 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.wG3z2ACvc3 2024-09-17T15:54:45.014 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.wG3z2ACvc3 is 232KB 2024-09-17T15:54:45.107 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.wG3z2ACvc3 2024-09-17T15:54:45.122 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/mon/ceph-g to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/data/mon.g.tgz 2024-09-17T15:54:45.123 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-09-17T15:54:45.178 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.Rd7nsiXl4V 2024-09-17T15:54:45.178 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-g -- . > /tmp/tmp.Rd7nsiXl4V 2024-09-17T15:54:45.276 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.Rd7nsiXl4V 2024-09-17T15:54:45.358 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.Rd7nsiXl4V is 229KB 2024-09-17T15:54:45.454 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.Rd7nsiXl4V 2024-09-17T15:54:45.469 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/mon/ceph-h to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/data/mon.h.tgz 2024-09-17T15:54:45.469 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-09-17T15:54:45.523 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.CHp4Azpu5R 2024-09-17T15:54:45.523 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-h -- . > /tmp/tmp.CHp4Azpu5R 2024-09-17T15:54:45.619 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.CHp4Azpu5R 2024-09-17T15:54:45.701 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.CHp4Azpu5R is 220KB 2024-09-17T15:54:45.798 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.CHp4Azpu5R 2024-09-17T15:54:45.813 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/mon/ceph-i to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/data/mon.i.tgz 2024-09-17T15:54:45.813 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-09-17T15:54:45.868 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.VnG9wnWmZH 2024-09-17T15:54:45.869 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-i -- . > /tmp/tmp.VnG9wnWmZH 2024-09-17T15:54:45.965 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.VnG9wnWmZH 2024-09-17T15:54:46.048 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.VnG9wnWmZH is 220KB 2024-09-17T15:54:46.141 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.VnG9wnWmZH 2024-09-17T15:54:46.156 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-17T15:54:46.156 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T15:54:46.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T15:54:46.246 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T15:54:46.274 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T15:54:46.276 INFO:tasks.ceph:Archiving crash dumps... 2024-09-17T15:54:46.277 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/remote/smithi064/crash 2024-09-17T15:54:46.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T15:54:46.316 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/remote/smithi100/crash 2024-09-17T15:54:46.317 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T15:54:46.340 INFO:tasks.ceph:Compressing logs... 2024-09-17T15:54:46.340 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T15:54:46.361 DEBUG:teuthology.orchestra.run.smithi100:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T15:54:46.386 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-17T15:54:46.386 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41700.log 2024-09-17T15:54:46.386 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38674.log 2024-09-17T15:54:46.525 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42120.log 2024-09-17T15:54:46.526 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35581.log 2024-09-17T15:54:46.526 INFO:teuthology.orchestra.run.smithi064.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41700.log.gz 2024-09-17T15:54:46.526 INFO:teuthology.orchestra.run.smithi064.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38674.log.gz 2024-09-17T15:54:46.526 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39103.log 2024-09-17T15:54:46.526 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35581.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42120.log.gz 2024-09-17T15:54:46.526 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41199.log 2024-09-17T15:54:46.526 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42204.log 2024-09-17T15:54:46.526 INFO:teuthology.orchestra.run.smithi064.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35581.log.gz 2024-09-17T15:54:46.526 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41199.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39103.log.gz 2024-09-17T15:54:46.526 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35916.log 2024-09-17T15:54:46.527 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42204.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41199.log.gz 2024-09-17T15:54:46.527 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40315.log 2024-09-17T15:54:46.527 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36169.log 2024-09-17T15:54:46.527 INFO:teuthology.orchestra.run.smithi064.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42204.log.gz 2024-09-17T15:54:46.527 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40315.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35916.log.gz 2024-09-17T15:54:46.527 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35657.log 2024-09-17T15:54:46.527 INFO:teuthology.orchestra.run.smithi064.stderr: 89.1%/var/log/ceph/ceph-client.admin.36169.log: -- replaced with /var/log/ceph/ceph-client.admin.40315.log.gz 2024-09-17T15:54:46.527 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38044.log 2024-09-17T15:54:46.527 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40989.log 2024-09-17T15:54:46.527 INFO:teuthology.orchestra.run.smithi064.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36169.log.gz 2024-09-17T15:54:46.527 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38044.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35657.log.gz 2024-09-17T15:54:46.527 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37664.log 2024-09-17T15:54:46.528 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41241.log 2024-09-17T15:54:46.528 INFO:teuthology.orchestra.run.smithi064.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38044.log.gz 2024-09-17T15:54:46.528 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.40989.log.gz 2024-09-17T15:54:46.528 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37706.log 2024-09-17T15:54:46.528 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37664.log: /var/log/ceph/ceph-client.admin.41241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35918.log 2024-09-17T15:54:46.528 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37664.log.gz 2024-09-17T15:54:46.528 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37706.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41241.log.gz 2024-09-17T15:54:46.528 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38296.log 2024-09-17T15:54:46.528 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39271.log 2024-09-17T15:54:46.528 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35918.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37706.log.gz 2024-09-17T15:54:46.528 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42036.log 2024-09-17T15:54:46.528 INFO:teuthology.orchestra.run.smithi064.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35918.log.gz 2024-09-17T15:54:46.529 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39271.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38296.log.gz 2024-09-17T15:54:46.529 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41157.log 2024-09-17T15:54:46.529 INFO:teuthology.orchestra.run.smithi064.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39271.log.gz 2024-09-17T15:54:46.529 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38590.log 2024-09-17T15:54:46.529 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41157.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42036.log.gz 2024-09-17T15:54:46.529 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40609.log 2024-09-17T15:54:46.529 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38977.log 2024-09-17T15:54:46.529 INFO:teuthology.orchestra.run.smithi064.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41157.log.gz 2024-09-17T15:54:46.529 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40609.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.38590.log.gz 2024-09-17T15:54:46.529 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-17T15:54:46.529 INFO:teuthology.orchestra.run.smithi064.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40609.log.gz 2024-09-17T15:54:46.529 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32839.log 2024-09-17T15:54:46.529 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37748.log 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38977.log.gz 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38464.log 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32839.log.gz 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40567.log 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37748.log.gz 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35525.log 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38464.log: /var/log/ceph/ceph-client.admin.40567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39650.log 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35525.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38464.log.gz 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35921.log 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40567.log.gz 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35525.log.gz 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38935.log 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39061.log 2024-09-17T15:54:46.530 INFO:teuthology.orchestra.run.smithi064.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39650.log.gz 2024-09-17T15:54:46.531 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35921.log: /var/log/ceph/ceph-client.admin.38935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37622.log 2024-09-17T15:54:46.531 INFO:teuthology.orchestra.run.smithi064.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35921.log.gz 2024-09-17T15:54:46.531 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39061.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38935.log.gz -5 2024-09-17T15:54:46.531 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39524.log 2024-09-17T15:54:46.531 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38128.log 2024-09-17T15:54:46.531 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37622.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39061.log.gz 2024-09-17T15:54:46.531 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36413.log 2024-09-17T15:54:46.531 INFO:teuthology.orchestra.run.smithi064.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37622.log.gz 2024-09-17T15:54:46.531 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35741.log 2024-09-17T15:54:46.531 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38128.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39524.log.gz 2024-09-17T15:54:46.531 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41952.log 2024-09-17T15:54:46.531 INFO:teuthology.orchestra.run.smithi064.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38128.log.gz 2024-09-17T15:54:46.532 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35741.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36413.log.gz 2024-09-17T15:54:46.532 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36497.log 2024-09-17T15:54:46.532 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41952.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35741.log.gz 2024-09-17T15:54:46.532 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39313.log 2024-09-17T15:54:46.532 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35699.log 2024-09-17T15:54:46.532 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36497.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41952.log.gz 2024-09-17T15:54:46.532 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38422.log 2024-09-17T15:54:46.532 INFO:teuthology.orchestra.run.smithi064.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36497.log.gz 2024-09-17T15:54:46.532 INFO:teuthology.orchestra.run.smithi064.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39313.log.gz 2024-09-17T15:54:46.532 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32766.log 2024-09-17T15:54:46.532 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35699.log: /var/log/ceph/ceph-client.admin.38422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39944.log 2024-09-17T15:54:46.533 INFO:teuthology.orchestra.run.smithi064.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35699.log.gz 2024-09-17T15:54:46.533 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32766.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38422.log.gz 2024-09-17T15:54:46.533 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40483.log 2024-09-17T15:54:46.533 INFO:teuthology.orchestra.run.smithi064.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32766.log.gz 2024-09-17T15:54:46.533 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38380.log 2024-09-17T15:54:46.533 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39860.log 2024-09-17T15:54:46.533 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40483.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39944.log.gz 2024-09-17T15:54:46.533 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39398.log 2024-09-17T15:54:46.533 INFO:teuthology.orchestra.run.smithi064.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40483.log.gz 2024-09-17T15:54:46.534 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35924.log 89.3% 2024-09-17T15:54:46.534 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38380.log.gz 2024-09-17T15:54:46.534 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39860.log: /var/log/ceph/ceph-client.admin.39398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41448.log 2024-09-17T15:54:46.534 INFO:teuthology.orchestra.run.smithi064.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39860.log.gz 2024-09-17T15:54:46.534 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42078.log 2024-09-17T15:54:46.534 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39398.log.gz 2024-09-17T15:54:46.534 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41742.log 2024-09-17T15:54:46.534 INFO:teuthology.orchestra.run.smithi064.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35924.log.gz 2024-09-17T15:54:46.534 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42246.log 2024-09-17T15:54:46.534 INFO:teuthology.orchestra.run.smithi064.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41448.log.gz 2024-09-17T15:54:46.534 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41742.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42078.log.gz 2024-09-17T15:54:46.534 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38338.log 2024-09-17T15:54:46.535 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41742.log.gz 2024-09-17T15:54:46.535 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37255.log 2024-09-17T15:54:46.535 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T15:54:46.535 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38338.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42246.log.gz 2024-09-17T15:54:46.535 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42540.log 2024-09-17T15:54:46.535 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38338.log.gz 2024-09-17T15:54:46.535 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42162.log 2024-09-17T15:54:46.535 INFO:teuthology.orchestra.run.smithi064.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37255.log.gz 2024-09-17T15:54:46.535 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40399.log 2024-09-17T15:54:46.535 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42162.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42540.log.gz 2024-09-17T15:54:46.535 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41031.log 2024-09-17T15:54:46.536 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36175.log 2024-09-17T15:54:46.536 INFO:teuthology.orchestra.run.smithi064.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42162.log.gz 2024-09-17T15:54:46.536 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36961.log 2024-09-17T15:54:46.536 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40399.log.gz 2024-09-17T15:54:46.536 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41574.log 2024-09-17T15:54:46.536 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41031.log.gz 2024-09-17T15:54:46.536 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39986.log 2024-09-17T15:54:46.536 INFO:teuthology.orchestra.run.smithi064.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36175.log.gz 2024-09-17T15:54:46.536 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41574.log: gzip 88.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.36961.log.gz --verbose 2024-09-17T15:54:46.536 INFO:teuthology.orchestra.run.smithi064.stderr: -- /var/log/ceph/ceph-client.admin.39440.log 2024-09-17T15:54:46.536 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39986.log: gzip -5 --verbose -- 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41574.log.gz 2024-09-17T15:54:46.536 INFO:teuthology.orchestra.run.smithi064.stderr: /var/log/ceph/ceph-client.admin.37832.log 2024-09-17T15:54:46.536 INFO:teuthology.orchestra.run.smithi064.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39986.log.gz 2024-09-17T15:54:46.537 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39440.log: 96.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T15:54:46.537 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40695.log 2024-09-17T15:54:46.537 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37832.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27760.log 2024-09-17T15:54:46.537 INFO:teuthology.orchestra.run.smithi064.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39440.log.gz 2024-09-17T15:54:46.537 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37003.log 2024-09-17T15:54:46.537 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37832.log.gz 2024-09-17T15:54:46.537 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.tmp-client.admin.27760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36623.log 2024-09-17T15:54:46.537 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27760.log.gz 2024-09-17T15:54:46.537 INFO:teuthology.orchestra.run.smithi064.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40695.log.gz 2024-09-17T15:54:46.537 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36539.log 2024-09-17T15:54:46.537 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36623.log: gzip 89.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.37003.log.gz 2024-09-17T15:54:46.538 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40525.log 2024-09-17T15:54:46.538 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38170.log 2024-09-17T15:54:46.538 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36539.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36623.log.gz 2024-09-17T15:54:46.538 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38212.log 2024-09-17T15:54:46.538 INFO:teuthology.orchestra.run.smithi064.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36539.log.gz 2024-09-17T15:54:46.538 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36176.log 2024-09-17T15:54:46.538 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38170.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40525.log.gz 2024-09-17T15:54:46.538 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38212.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-17T15:54:46.538 INFO:teuthology.orchestra.run.smithi064.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38170.log.gz 2024-09-17T15:54:46.538 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41868.log 2024-09-17T15:54:46.538 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36176.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38212.log.gz 2024-09-17T15:54:46.538 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40737.log 2024-09-17T15:54:46.539 INFO:teuthology.orchestra.run.smithi064.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36176.log.gz 2024-09-17T15:54:46.539 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37045.log 2024-09-17T15:54:46.539 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41868.log: /var/log/ceph/ceph-client.admin.40737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42582.log 2024-09-17T15:54:46.539 INFO:teuthology.orchestra.run.smithi064.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41868.log.gz 2024-09-17T15:54:46.539 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37213.log 2024-09-17T15:54:46.539 INFO:teuthology.orchestra.run.smithi064.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40737.log.gz 2024-09-17T15:54:46.539 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39776.log 2024-09-17T15:54:46.539 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37045.log.gz 2024-09-17T15:54:46.539 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39566.log 2024-09-17T15:54:46.539 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37213.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42582.log.gz 2024-09-17T15:54:46.539 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36835.log 2024-09-17T15:54:46.539 INFO:teuthology.orchestra.run.smithi064.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37213.log.gz 2024-09-17T15:54:46.539 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39566.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39776.log.gz 2024-09-17T15:54:46.540 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36329.log 2024-09-17T15:54:46.540 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41490.log 2024-09-17T15:54:46.540 INFO:teuthology.orchestra.run.smithi064.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39566.log.gz 2024-09-17T15:54:46.540 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36329.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36835.log.gz 2024-09-17T15:54:46.540 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41910.log 2024-09-17T15:54:46.540 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35923.log 2024-09-17T15:54:46.540 INFO:teuthology.orchestra.run.smithi064.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36329.log.gz 2024-09-17T15:54:46.540 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41658.log 2024-09-17T15:54:46.540 INFO:teuthology.orchestra.run.smithi064.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41490.log.gz 2024-09-17T15:54:46.540 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35923.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41910.log.gz 2024-09-17T15:54:46.540 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42456.log 2024-09-17T15:54:46.540 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39187.log 2024-09-17T15:54:46.541 INFO:teuthology.orchestra.run.smithi064.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35923.log.gz 2024-09-17T15:54:46.541 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36581.log 2024-09-17T15:54:46.541 INFO:teuthology.orchestra.run.smithi064.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41658.log.gz 2024-09-17T15:54:46.541 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35327.log 2024-09-17T15:54:46.541 INFO:teuthology.orchestra.run.smithi064.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42456.log.gz 2024-09-17T15:54:46.541 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39734.log 2024-09-17T15:54:46.541 INFO:teuthology.orchestra.run.smithi064.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39187.log.gz 2024-09-17T15:54:46.541 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39019.log 2024-09-17T15:54:46.541 INFO:teuthology.orchestra.run.smithi064.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36581.log.gz 2024-09-17T15:54:46.541 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39692.log 2024-09-17T15:54:46.541 INFO:teuthology.orchestra.run.smithi064.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35327.log.gz 2024-09-17T15:54:46.541 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39608.log 2024-09-17T15:54:46.542 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39734.log.gz 2024-09-17T15:54:46.542 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39692.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39019.log.gz 2024-09-17T15:54:46.542 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-17T15:54:46.542 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.e.log 2024-09-17T15:54:46.542 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39608.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39692.log.gz 2024-09-17T15:54:46.542 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36709.log 2024-09-17T15:54:46.542 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39608.log.gz 2024-09-17T15:54:46.542 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37297.log 2024-09-17T15:54:46.542 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37087.log 2024-09-17T15:54:46.542 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37297.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36709.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41532.log 2024-09-17T15:54:46.542 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:46.542 INFO:teuthology.orchestra.run.smithi064.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37297.log.gz 2024-09-17T15:54:46.543 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40905.log 2024-09-17T15:54:46.543 INFO:teuthology.orchestra.run.smithi064.stderr: 88.3%/var/log/ceph/ceph-client.admin.41532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40028.log 2024-09-17T15:54:46.543 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37087.log.gz 2024-09-17T15:54:46.543 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40905.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41532.log.gz 2024-09-17T15:54:46.543 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40273.log 2024-09-17T15:54:46.543 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40905.log.gz 2024-09-17T15:54:46.543 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39145.log 2024-09-17T15:54:46.543 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40028.log: /var/log/ceph/ceph-client.admin.40273.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40028.log.gz 2024-09-17T15:54:46.543 INFO:teuthology.orchestra.run.smithi064.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40273.log.gz 2024-09-17T15:54:46.543 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41283.log 2024-09-17T15:54:46.543 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40653.log 2024-09-17T15:54:46.543 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41283.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-17T15:54:46.543 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39145.log.gz 2024-09-17T15:54:46.544 INFO:teuthology.orchestra.run.smithi064.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41283.log.gz 2024-09-17T15:54:46.544 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40653.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40653.log.gz 2024-09-17T15:54:46.544 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40947.log 2024-09-17T15:54:46.544 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37916.log 2024-09-17T15:54:46.544 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32415.log 2024-09-17T15:54:46.544 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37916.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40947.log.gz 2024-09-17T15:54:46.544 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39482.log 2024-09-17T15:54:46.544 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32415.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37916.log.gz 2024-09-17T15:54:46.544 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35925.log 2024-09-17T15:54:46.544 INFO:teuthology.orchestra.run.smithi064.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32415.log.gz 2024-09-17T15:54:46.544 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38548.log 2024-09-17T15:54:46.544 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35925.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39482.log.gz 2024-09-17T15:54:46.544 INFO:teuthology.orchestra.run.smithi064.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35925.log.gz 2024-09-17T15:54:46.545 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40357.log 2024-09-17T15:54:46.545 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37874.log 2024-09-17T15:54:46.545 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38548.log: /var/log/ceph/ceph-client.admin.40357.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38548.log.gz 2024-09-17T15:54:46.545 INFO:teuthology.orchestra.run.smithi064.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38086.log 2024-09-17T15:54:46.545 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40357.log.gz 2024-09-17T15:54:46.545 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41073.log 2024-09-17T15:54:46.545 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37874.log: /var/log/ceph/ceph-client.admin.38086.log: 88.9% 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38086.log.gz 2024-09-17T15:54:46.545 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37874.log.gz 2024-09-17T15:54:46.545 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T15:54:46.545 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38506.log 2024-09-17T15:54:46.545 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41073.log.gz 2024-09-17T15:54:46.545 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log 2024-09-17T15:54:46.545 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38506.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38506.log.gz 2024-09-17T15:54:46.546 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39902.log 2024-09-17T15:54:46.546 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40109.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40109.log.gz 2024-09-17T15:54:46.546 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37383.log 2024-09-17T15:54:46.546 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39902.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39902.log.gz 2024-09-17T15:54:46.546 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39356.log 2024-09-17T15:54:46.546 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37383.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37383.log.gz 2024-09-17T15:54:46.546 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40863.log 2024-09-17T15:54:46.546 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39356.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39356.log.gz 2024-09-17T15:54:46.546 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42414.log 2024-09-17T15:54:46.546 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40863.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40863.log.gz 2024-09-17T15:54:46.546 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42288.log 2024-09-17T15:54:46.546 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42414.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42414.log.gz 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40821.log 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42288.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42288.log.gz 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38002.log 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40821.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40821.log.gz 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42498.log 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38002.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38002.log.gz 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39229.log 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42498.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42498.log.gz 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.d.log 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39229.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39229.log.gz 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41826.log 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.d.log: 94.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42706.log 2024-09-17T15:54:46.547 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41826.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41826.log.gz 2024-09-17T15:54:46.548 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37339.log 2024-09-17T15:54:46.548 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42706.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42706.log.gz 2024-09-17T15:54:46.548 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36455.log 2024-09-17T15:54:46.548 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37339.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37339.log.gz 2024-09-17T15:54:46.548 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36793.log 2024-09-17T15:54:46.548 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36455.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36455.log.gz 2024-09-17T15:54:46.548 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37171.log 2024-09-17T15:54:46.548 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36793.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36793.log.gz 2024-09-17T15:54:46.548 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37958.log 2024-09-17T15:54:46.548 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37171.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37171.log.gz 2024-09-17T15:54:46.548 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40779.log 2024-09-17T15:54:46.548 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37958.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37958.log.gz 2024-09-17T15:54:46.548 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41616.log 2024-09-17T15:54:46.549 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40779.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40779.log.gz 2024-09-17T15:54:46.549 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-17T15:54:46.549 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41616.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41616.log.gz 2024-09-17T15:54:46.549 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35606.log 2024-09-17T15:54:46.549 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36177.log 2024-09-17T15:54:46.549 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35606.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35606.log.gz 2024-09-17T15:54:46.549 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41994.log 2024-09-17T15:54:46.549 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36177.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36177.log.gz 2024-09-17T15:54:46.549 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35783.log 2024-09-17T15:54:46.549 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41994.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41994.log.gz 2024-09-17T15:54:46.553 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36751.log 2024-09-17T15:54:46.553 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35783.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35783.log.gz 2024-09-17T15:54:46.553 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36665.log 2024-09-17T15:54:46.553 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36751.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36751.log.gz 2024-09-17T15:54:46.558 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37580.log 2024-09-17T15:54:46.572 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36665.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36665.log.gz 2024-09-17T15:54:46.572 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36166.log 2024-09-17T15:54:46.572 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37580.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37580.log.gz 2024-09-17T15:54:46.585 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37790.log 2024-09-17T15:54:46.591 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38632.log 2024-09-17T15:54:46.597 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37790.log: 88.9% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36166.log.gz 2024-09-17T15:54:46.597 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36371.log 2024-09-17T15:54:46.602 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38632.log: -- replaced with /var/log/ceph/ceph-client.admin.37790.log.gz 2024-09-17T15:54:46.615 INFO:teuthology.orchestra.run.smithi064.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38632.log.gz 2024-09-17T15:54:46.622 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36877.log 2024-09-17T15:54:46.622 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36371.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36371.log.gz 2024-09-17T15:54:46.631 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39740.log 2024-09-17T15:54:46.631 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39850.log 2024-09-17T15:54:46.631 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35074.log 2024-09-17T15:54:46.631 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39740.log: /var/log/ceph/ceph-client.admin.39850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37421.log 2024-09-17T15:54:46.631 INFO:teuthology.orchestra.run.smithi100.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39740.log.gz 2024-09-17T15:54:46.631 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40390.log 2024-09-17T15:54:46.631 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39371.log 2024-09-17T15:54:46.632 INFO:teuthology.orchestra.run.smithi100.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39850.log.gz 2024-09-17T15:54:46.632 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40390.log: gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.40065.log -- replaced with /var/log/ceph/ceph-client.admin.35074.log.gz 2024-09-17T15:54:46.632 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:54:46.632 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39810.log 2024-09-17T15:54:46.632 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39371.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37421.log.gz 2024-09-17T15:54:46.632 INFO:teuthology.orchestra.run.smithi100.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40390.log.gz 2024-09-17T15:54:46.632 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40136.log 2024-09-17T15:54:46.632 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39810.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40065.log.gz 2024-09-17T15:54:46.632 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40066.log 2024-09-17T15:54:46.632 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40136.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.39371.log.gz 2024-09-17T15:54:46.632 INFO:teuthology.orchestra.run.smithi100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38944.log 2024-09-17T15:54:46.633 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36036.log 2024-09-17T15:54:46.633 INFO:teuthology.orchestra.run.smithi100.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40136.log.gz 2024-09-17T15:54:46.633 INFO:teuthology.orchestra.run.smithi100.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39810.log.gz 2024-09-17T15:54:46.633 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36777.log 2024-09-17T15:54:46.633 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36036.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40066.log.gz 2024-09-17T15:54:46.633 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36242.log 2024-09-17T15:54:46.633 INFO:teuthology.orchestra.run.smithi100.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38944.log.gz 2024-09-17T15:54:46.633 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36777.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36036.log.gz 2024-09-17T15:54:46.633 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39050.log 2024-09-17T15:54:46.633 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35692.log 2024-09-17T15:54:46.633 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39050.log: gzip 87.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.36242.log.gz 2024-09-17T15:54:46.633 INFO:teuthology.orchestra.run.smithi100.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39158.log 2024-09-17T15:54:46.633 INFO:teuthology.orchestra.run.smithi100.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36777.log.gz 2024-09-17T15:54:46.634 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38444.log 2024-09-17T15:54:46.634 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38443.log 2024-09-17T15:54:46.634 INFO:teuthology.orchestra.run.smithi100.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39050.log.gz 2024-09-17T15:54:46.634 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39811.log 2024-09-17T15:54:46.634 INFO:teuthology.orchestra.run.smithi100.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39158.log.gz 2024-09-17T15:54:46.634 INFO:teuthology.orchestra.run.smithi100.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35692.log.gz 2024-09-17T15:54:46.634 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36884.log 2024-09-17T15:54:46.634 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39811.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38444.log.gz 2024-09-17T15:54:46.634 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35830.log 2024-09-17T15:54:46.634 INFO:teuthology.orchestra.run.smithi100.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38443.log.gz 2024-09-17T15:54:46.634 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38551.log 2024-09-17T15:54:46.634 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35830.log: gzip 90.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.39811.log.gz --verbose 2024-09-17T15:54:46.634 INFO:teuthology.orchestra.run.smithi100.stderr: -- /var/log/ceph/ceph-mon.g.log 2024-09-17T15:54:46.635 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39265.log 2024-09-17T15:54:46.635 INFO:teuthology.orchestra.run.smithi100.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36884.log.gz 2024-09-17T15:54:46.635 INFO:teuthology.orchestra.run.smithi100.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35830.log.gz 2024-09-17T15:54:46.635 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-mon.g.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39516.log 2024-09-17T15:54:46.635 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38551.log.gz 2024-09-17T15:54:46.635 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32031.log 2024-09-17T15:54:46.635 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38192.log 2024-09-17T15:54:46.635 INFO:teuthology.orchestra.run.smithi100.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39265.log.gz 2024-09-17T15:54:46.635 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.32031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37848.log 2024-09-17T15:54:46.635 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38192.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32031.log.gz 2024-09-17T15:54:46.635 INFO:teuthology.orchestra.run.smithi100.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39516.log.gz 2024-09-17T15:54:46.635 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38872.log 2024-09-17T15:54:46.635 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36955.log 2024-09-17T15:54:46.636 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37100.log 2024-09-17T15:54:46.636 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38192.log.gz 2024-09-17T15:54:46.636 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38872.log: /var/log/ceph/ceph-client.admin.36955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35795.log 2024-09-17T15:54:46.636 INFO:teuthology.orchestra.run.smithi100.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37848.log.gz 2024-09-17T15:54:46.636 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36310.log 2024-09-17T15:54:46.636 INFO:teuthology.orchestra.run.smithi100.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38872.log.gz 2024-09-17T15:54:46.636 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37635.log 2024-09-17T15:54:46.636 INFO:teuthology.orchestra.run.smithi100.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36955.log.gz 2024-09-17T15:54:46.636 INFO:teuthology.orchestra.run.smithi100.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37100.log.gz 2024-09-17T15:54:46.636 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34972.log 2024-09-17T15:54:46.636 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39478.log 2024-09-17T15:54:46.636 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.34972.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35795.log.gz 2024-09-17T15:54:46.637 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36002.log 2024-09-17T15:54:46.637 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39478.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37635.log.gz 2024-09-17T15:54:46.637 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36310.log.gz 2024-09-17T15:54:46.637 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39300.log 2024-09-17T15:54:46.637 INFO:teuthology.orchestra.run.smithi100.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34972.log.gz 2024-09-17T15:54:46.637 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38979.log 2024-09-17T15:54:46.637 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36002.log: /var/log/ceph/ceph-client.admin.39300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37597.log 2024-09-17T15:54:46.637 INFO:teuthology.orchestra.run.smithi100.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36002.log.gz 2024-09-17T15:54:46.637 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39478.log.gz 2024-09-17T15:54:46.637 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37314.log 2024-09-17T15:54:46.637 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37597.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39300.log.gzgzip 2024-09-17T15:54:46.637 INFO:teuthology.orchestra.run.smithi100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36741.log 2024-09-17T15:54:46.637 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36413.log 2024-09-17T15:54:46.638 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36741.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37597.log.gz 2024-09-17T15:54:46.638 INFO:teuthology.orchestra.run.smithi100.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38979.log.gz 2024-09-17T15:54:46.638 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38229.log 2024-09-17T15:54:46.638 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38407.log 2024-09-17T15:54:46.638 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36413.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37314.log.gz 2024-09-17T15:54:46.638 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35521.log 2024-09-17T15:54:46.638 INFO:teuthology.orchestra.run.smithi100.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36741.log.gz 2024-09-17T15:54:46.638 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38407.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38229.log.gz 2024-09-17T15:54:46.638 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38658.log 2024-09-17T15:54:46.638 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37811.log 2024-09-17T15:54:46.638 INFO:teuthology.orchestra.run.smithi100.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38407.log.gz 2024-09-17T15:54:46.638 INFO:teuthology.orchestra.run.smithi100.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36413.log.gz 2024-09-17T15:54:46.638 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39514.log 2024-09-17T15:54:46.639 INFO:teuthology.orchestra.run.smithi100.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35521.log.gz 2024-09-17T15:54:46.639 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36849.log 2024-09-17T15:54:46.639 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39514.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38658.log.gz 2024-09-17T15:54:46.639 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37526.log 2024-09-17T15:54:46.639 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39515.log 2024-09-17T15:54:46.639 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36849.log: 87.6%/var/log/ceph/ceph-client.admin.37526.log: -- replaced with /var/log/ceph/ceph-client.admin.37811.log.gz 2024-09-17T15:54:46.639 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40135.log 2024-09-17T15:54:46.639 INFO:teuthology.orchestra.run.smithi100.stderr: 88.4% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39514.log.gz 2024-09-17T15:54:46.639 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36849.log.gz 2024-09-17T15:54:46.639 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35589.log 2024-09-17T15:54:46.639 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40135.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37526.log.gz 2024-09-17T15:54:46.639 INFO:teuthology.orchestra.run.smithi100.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39515.log.gz 2024-09-17T15:54:46.639 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36345.log 2024-09-17T15:54:46.640 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37207.log 2024-09-17T15:54:46.640 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35589.log: /var/log/ceph/ceph-client.admin.36345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36885.log 2024-09-17T15:54:46.640 INFO:teuthology.orchestra.run.smithi100.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40135.log.gz 2024-09-17T15:54:46.640 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35280.log 2024-09-17T15:54:46.640 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36345.log.gz 2024-09-17T15:54:46.640 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38335.log 2024-09-17T15:54:46.640 INFO:teuthology.orchestra.run.smithi100.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35589.log.gz 2024-09-17T15:54:46.640 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38766.log 2024-09-17T15:54:46.640 INFO:teuthology.orchestra.run.smithi100.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37207.log.gz 2024-09-17T15:54:46.640 INFO:teuthology.orchestra.run.smithi100.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36885.log.gz 2024-09-17T15:54:46.640 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38657.log 2024-09-17T15:54:46.641 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38766.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35280.log.gz 2024-09-17T15:54:46.641 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36566.log 2024-09-17T15:54:46.641 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38015.log 2024-09-17T15:54:46.641 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38657.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38335.log.gz 2024-09-17T15:54:46.641 INFO:teuthology.orchestra.run.smithi100.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38766.log.gz 2024-09-17T15:54:46.641 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39409.log 2024-09-17T15:54:46.641 INFO:teuthology.orchestra.run.smithi100.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38657.log.gz 2024-09-17T15:54:46.641 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35315.log 2024-09-17T15:54:46.641 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38015.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36566.log.gz 2024-09-17T15:54:46.641 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40172.log 2024-09-17T15:54:46.641 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35315.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38015.log.gz 2024-09-17T15:54:46.641 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36565.log 2024-09-17T15:54:46.642 INFO:teuthology.orchestra.run.smithi100.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39409.log.gz 2024-09-17T15:54:46.642 INFO:teuthology.orchestra.run.smithi100.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35315.log.gz 2024-09-17T15:54:46.642 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38981.log 2024-09-17T15:54:46.642 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40172.log: /var/log/ceph/ceph-client.admin.36565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39741.log 2024-09-17T15:54:46.642 INFO:teuthology.orchestra.run.smithi100.stderr: 87.1%/var/log/ceph/ceph-client.admin.38981.log: -- replaced with /var/log/ceph/ceph-client.admin.40172.log.gz 2024-09-17T15:54:46.642 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38620.log 2024-09-17T15:54:46.642 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38121.log 2024-09-17T15:54:46.642 INFO:teuthology.orchestra.run.smithi100.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36565.log.gz 2024-09-17T15:54:46.642 INFO:teuthology.orchestra.run.smithi100.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38981.log.gz 2024-09-17T15:54:46.642 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40279.log 2024-09-17T15:54:46.642 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38656.log 2024-09-17T15:54:46.642 INFO:teuthology.orchestra.run.smithi100.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39741.log.gz 2024-09-17T15:54:46.643 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38121.log: /var/log/ceph/ceph-client.admin.40279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35796.log 2024-09-17T15:54:46.643 INFO:teuthology.orchestra.run.smithi100.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38620.log.gz 2024-09-17T15:54:46.643 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38656.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-17T15:54:46.643 INFO:teuthology.orchestra.run.smithi100.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38121.log.gz 2024-09-17T15:54:46.643 INFO:teuthology.orchestra.run.smithi100.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40279.log.gz 2024-09-17T15:54:46.643 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35727.log 2024-09-17T15:54:46.643 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40391.log 2024-09-17T15:54:46.643 INFO:teuthology.orchestra.run.smithi100.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35796.log.gz 2024-09-17T15:54:46.643 INFO:teuthology.orchestra.run.smithi100.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38656.log.gz 2024-09-17T15:54:46.643 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38728.log 2024-09-17T15:54:46.643 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40391.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35727.log.gz 2024-09-17T15:54:46.643 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32151.log 2024-09-17T15:54:46.643 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37849.log 2024-09-17T15:54:46.644 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38728.log: /var/log/ceph/ceph-client.admin.32151.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40391.log.gz 2024-09-17T15:54:46.644 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36346.log 2024-09-17T15:54:46.644 INFO:teuthology.orchestra.run.smithi100.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38728.log.gz 2024-09-17T15:54:46.644 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32151.log.gz 2024-09-17T15:54:46.644 INFO:teuthology.orchestra.run.smithi100.stderr:gzip/var/log/ceph/ceph-client.admin.37849.log: -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T15:54:46.644 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38764.log 2024-09-17T15:54:46.644 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37277.log 2024-09-17T15:54:46.644 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37849.log.gz 2024-09-17T15:54:46.644 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36346.log.gz 2024-09-17T15:54:46.644 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37206.log 2024-09-17T15:54:46.644 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39301.log 2024-09-17T15:54:46.644 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37277.log: /var/log/ceph/ceph-client.admin.37206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35108.log 2024-09-17T15:54:46.644 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38764.log.gz 2024-09-17T15:54:46.645 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37277.log.gz 2024-09-17T15:54:46.645 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39301.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37206.log.gz 2024-09-17T15:54:46.645 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39157.log 2024-09-17T15:54:46.645 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35108.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39301.log.gz 2024-09-17T15:54:46.645 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32111.log 2024-09-17T15:54:46.645 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39851.log 2024-09-17T15:54:46.645 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.32111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36035.log 2024-09-17T15:54:46.645 INFO:teuthology.orchestra.run.smithi100.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35108.log.gz 2024-09-17T15:54:46.645 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38228.log 2024-09-17T15:54:46.645 INFO:teuthology.orchestra.run.smithi100.stderr: 87.8% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39157.log.gz 2024-09-17T15:54:46.645 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32111.log.gz 2024-09-17T15:54:46.645 INFO:teuthology.orchestra.run.smithi100.stderr: 96.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T15:54:46.646 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36778.log 2024-09-17T15:54:46.646 INFO:teuthology.orchestra.run.smithi100.stderr: 86.9%/var/log/ceph/ceph-client.admin.36035.log: -- replaced with /var/log/ceph/ceph-client.admin.39851.log.gz 2024-09-17T15:54:46.646 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37528.log 2024-09-17T15:54:46.646 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37740.log 2024-09-17T15:54:46.646 INFO:teuthology.orchestra.run.smithi100.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36035.log.gz 2024-09-17T15:54:46.646 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37528.log: gzip -5 --verbose -- 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36778.log.gz 2024-09-17T15:54:46.646 INFO:teuthology.orchestra.run.smithi100.stderr: /var/log/ceph/ceph-client.admin.32197.log 2024-09-17T15:54:46.646 INFO:teuthology.orchestra.run.smithi100.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38228.log.gz 2024-09-17T15:54:46.646 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37312.log 2024-09-17T15:54:46.646 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37528.log.gz 2024-09-17T15:54:46.646 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.32197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38621.log 2024-09-17T15:54:46.646 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37312.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32197.log.gz 2024-09-17T15:54:46.646 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37740.log.gz 2024-09-17T15:54:46.647 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40352.log 2024-09-17T15:54:46.647 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37634.log 2024-09-17T15:54:46.647 INFO:teuthology.orchestra.run.smithi100.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38621.log.gz 2024-09-17T15:54:46.647 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37919.log 2024-09-17T15:54:46.647 INFO:teuthology.orchestra.run.smithi100.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37312.log.gz 2024-09-17T15:54:46.647 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39051.log 2024-09-17T15:54:46.647 INFO:teuthology.orchestra.run.smithi100.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37634.log.gz 2024-09-17T15:54:46.647 INFO:teuthology.orchestra.run.smithi100.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40352.log.gz 2024-09-17T15:54:46.647 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35522.log 2024-09-17T15:54:46.647 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37276.log 2024-09-17T15:54:46.647 INFO:teuthology.orchestra.run.smithi100.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37919.log.gz 2024-09-17T15:54:46.647 INFO:teuthology.orchestra.run.smithi100.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39051.log.gz 2024-09-17T15:54:46.648 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35625.log 2024-09-17T15:54:46.648 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39408.log 2024-09-17T15:54:46.648 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37276.log: /var/log/ceph/ceph-client.admin.35625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37527.log 2024-09-17T15:54:46.648 INFO:teuthology.orchestra.run.smithi100.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35522.log.gz 2024-09-17T15:54:46.648 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37812.log 2024-09-17T15:54:46.648 INFO:teuthology.orchestra.run.smithi100.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37276.log.gz 2024-09-17T15:54:46.648 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37527.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35625.log.gz 2024-09-17T15:54:46.648 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35007.log 2024-09-17T15:54:46.648 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39408.log.gz 2024-09-17T15:54:46.648 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35899.log 2024-09-17T15:54:46.648 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37527.log.gz 2024-09-17T15:54:46.648 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35007.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37812.log.gz 2024-09-17T15:54:46.648 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39739.log 2024-09-17T15:54:46.649 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40029.log 2024-09-17T15:54:46.649 INFO:teuthology.orchestra.run.smithi100.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35007.log.gz 2024-09-17T15:54:46.649 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39739.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39194.log 2024-09-17T15:54:46.649 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35899.log.gz 2024-09-17T15:54:46.649 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36567.log 2024-09-17T15:54:46.649 INFO:teuthology.orchestra.run.smithi100.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40029.log.gz 2024-09-17T15:54:46.649 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39194.log: gzip 87.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39739.log.gz -- 2024-09-17T15:54:46.649 INFO:teuthology.orchestra.run.smithi100.stderr: /var/log/ceph/ceph-client.admin.38442.log 2024-09-17T15:54:46.649 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39193.log 2024-09-17T15:54:46.649 INFO:teuthology.orchestra.run.smithi100.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39194.log.gz 2024-09-17T15:54:46.649 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38122.log 2024-09-17T15:54:46.649 INFO:teuthology.orchestra.run.smithi100.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36567.log.gz 2024-09-17T15:54:46.650 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37704.log 2024-09-17T15:54:46.650 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35384.log 2024-09-17T15:54:46.650 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38122.log: /var/log/ceph/ceph-client.admin.37704.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38442.log.gz 2024-09-17T15:54:46.650 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38513.log 2024-09-17T15:54:46.650 INFO:teuthology.orchestra.run.smithi100.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38122.log.gz 2024-09-17T15:54:46.650 INFO:teuthology.orchestra.run.smithi100.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39193.log.gz 2024-09-17T15:54:46.650 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38943.log 2024-09-17T15:54:46.650 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38513.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37704.log.gz 2024-09-17T15:54:46.650 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34971.log 2024-09-17T15:54:46.650 INFO:teuthology.orchestra.run.smithi100.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35384.log.gz 2024-09-17T15:54:46.650 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35624.log 2024-09-17T15:54:46.650 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.34971.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38513.log.gz 2024-09-17T15:54:46.651 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35006.log 2024-09-17T15:54:46.651 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37490.log 2024-09-17T15:54:46.651 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38943.log.gz 2024-09-17T15:54:46.651 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35624.log.gz 2024-09-17T15:54:46.651 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39302.log 2024-09-17T15:54:46.651 INFO:teuthology.orchestra.run.smithi100.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34971.log.gz 2024-09-17T15:54:46.651 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37705.log 2024-09-17T15:54:46.651 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35006.log.gz 2024-09-17T15:54:46.651 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40242.log 2024-09-17T15:54:46.651 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40389.log 2024-09-17T15:54:46.651 INFO:teuthology.orchestra.run.smithi100.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37490.log.gz 2024-09-17T15:54:46.651 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38299.log/var/log/ceph/ceph-client.admin.40242.log: 88.5% 2024-09-17T15:54:46.651 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39302.log.gz 2024-09-17T15:54:46.652 INFO:teuthology.orchestra.run.smithi100.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37705.log.gz 2024-09-17T15:54:46.652 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35728.log 2024-09-17T15:54:46.652 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37098.log 2024-09-17T15:54:46.652 INFO:teuthology.orchestra.run.smithi100.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40242.log.gz 2024-09-17T15:54:46.652 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36991.log 2024-09-17T15:54:46.652 INFO:teuthology.orchestra.run.smithi100.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40389.log.gz 2024-09-17T15:54:46.652 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35178.log 2024-09-17T15:54:46.652 INFO:teuthology.orchestra.run.smithi100.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38299.log.gz 2024-09-17T15:54:46.652 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35486.log 2024-09-17T15:54:46.652 INFO:teuthology.orchestra.run.smithi100.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35728.log.gz 2024-09-17T15:54:46.652 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40278.log 2024-09-17T15:54:46.652 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37098.log.gz 2024-09-17T15:54:46.652 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36138.log 2024-09-17T15:54:46.653 INFO:teuthology.orchestra.run.smithi100.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35178.log.gz 2024-09-17T15:54:46.653 INFO:teuthology.orchestra.run.smithi100.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36991.log.gz 2024-09-17T15:54:46.653 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40278.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.h.log 2024-09-17T15:54:46.653 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.i.log 2024-09-17T15:54:46.653 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36138.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35486.log.gz 2024-09-17T15:54:46.653 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-mon.h.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35212.log 2024-09-17T15:54:46.653 INFO:teuthology.orchestra.run.smithi100.stderr: 88.1%/var/log/ceph/ceph-mon.i.log: -- replaced with /var/log/ceph/ceph-client.admin.40278.log.gz 2024-09-17T15:54:46.653 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose 88.1% -- -- replaced with /var/log/ceph/ceph-client.admin.36138.log.gz /var/log/ceph/ceph-client.admin.35693.log 2024-09-17T15:54:46.653 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:54:46.653 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39195.log 2024-09-17T15:54:46.653 INFO:teuthology.orchestra.run.smithi100.stderr:gzip/var/log/ceph/ceph-client.admin.35693.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36670.log 2024-09-17T15:54:46.653 INFO:teuthology.orchestra.run.smithi100.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35212.log.gz 2024-09-17T15:54:46.654 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37205.log 2024-09-17T15:54:46.654 INFO:teuthology.orchestra.run.smithi100.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35693.log.gz 2024-09-17T15:54:46.654 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38765.log 2024-09-17T15:54:46.654 INFO:teuthology.orchestra.run.smithi100.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39195.log.gz 2024-09-17T15:54:46.654 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5/var/log/ceph/ceph-client.admin.37205.log: --verbose -- /var/log/ceph/ceph-client.admin.36742.log 2024-09-17T15:54:46.654 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38765.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36670.log.gz 2024-09-17T15:54:46.654 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40353.log 2024-09-17T15:54:46.654 INFO:teuthology.orchestra.run.smithi100.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38765.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40028.log 2024-09-17T15:54:46.654 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:54:46.654 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36742.log: 88.9%/var/log/ceph/ceph-client.admin.40353.log: -- replaced with /var/log/ceph/ceph-client.admin.37205.log.gz 2024-09-17T15:54:46.654 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36886.log 2024-09-17T15:54:46.654 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40028.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40353.log.gz 2024-09-17T15:54:46.654 INFO:teuthology.orchestra.run.smithi100.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36742.log.gz 2024-09-17T15:54:46.655 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40173.log 2024-09-17T15:54:46.655 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35623.log 2024-09-17T15:54:46.655 INFO:teuthology.orchestra.run.smithi100.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40028.log.gz 2024-09-17T15:54:46.655 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39586.log 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36886.log.gz 2024-09-17T15:54:46.655 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:54:46.655 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37063.log 2024-09-17T15:54:46.655 INFO:teuthology.orchestra.run.smithi100.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40173.log.gz 2024-09-17T15:54:46.655 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38549.log 2024-09-17T15:54:46.655 INFO:teuthology.orchestra.run.smithi100.stderr: 88.3%/var/log/ceph/ceph-client.admin.37063.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39586.log.gz 2024-09-17T15:54:46.655 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35623.log.gz 2024-09-17T15:54:46.655 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38550.log 2024-09-17T15:54:46.655 INFO:teuthology.orchestra.run.smithi100.stderr:gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37063.log.gz 2024-09-17T15:54:46.655 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38549.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38980.log 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35213.log 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38980.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38550.log.gz 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr:gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38549.log.gz 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35831.log 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35213.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38980.log.gz 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38336.log 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36241.log 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35213.log.gz 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35590.log 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38336.log: /var/log/ceph/ceph-client.admin.36241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36672.log 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35831.log.gz 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38336.log.gz 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35005.log 2024-09-17T15:54:46.656 INFO:teuthology.orchestra.run.smithi100.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35590.log.gz 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36241.log.gz 2024-09-17T15:54:46.657 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:54:46.657 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36139.log 2024-09-17T15:54:46.657 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35005.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35520.log 2024-09-17T15:54:46.657 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36672.log.gz 2024-09-17T15:54:46.657 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39852.log 2024-09-17T15:54:46.657 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39088.log 2024-09-17T15:54:46.657 INFO:teuthology.orchestra.run.smithi100.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35005.log.gz 2024-09-17T15:54:46.657 INFO:teuthology.orchestra.run.smithi100.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36139.log.gz 2024-09-17T15:54:46.657 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38086.log 2024-09-17T15:54:46.657 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39088.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35520.log.gz 2024-09-17T15:54:46.657 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35110.log 2024-09-17T15:54:46.658 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38013.log 2024-09-17T15:54:46.658 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39852.log.gz 2024-09-17T15:54:46.658 INFO:teuthology.orchestra.run.smithi100.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39088.log.gz 2024-09-17T15:54:46.658 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35110.log: gzip -5 --verbose 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38086.log.gz 2024-09-17T15:54:46.658 INFO:teuthology.orchestra.run.smithi100.stderr: -- /var/log/ceph/ceph-client.admin.35419.log 2024-09-17T15:54:46.658 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38013.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-17T15:54:46.658 INFO:teuthology.orchestra.run.smithi100.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35110.log.gz 2024-09-17T15:54:46.658 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38836.log 2024-09-17T15:54:46.658 INFO:teuthology.orchestra.run.smithi100.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38013.log.gz 2024-09-17T15:54:46.658 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose 88.7% -- /var/log/ceph/ceph-client.admin.36634.log 2024-09-17T15:54:46.658 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35419.log.gz 2024-09-17T15:54:46.658 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38836.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T15:54:46.658 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35211.log 2024-09-17T15:54:46.659 INFO:teuthology.orchestra.run.smithi100.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38836.log.gz 2024-09-17T15:54:46.659 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37313.log 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36634.log.gz 2024-09-17T15:54:46.659 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:54:46.659 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37383.log 2024-09-17T15:54:46.659 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37313.log: 87.1% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37313.log.gz 2024-09-17T15:54:46.659 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35211.log.gz 2024-09-17T15:54:46.659 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35898.log 2024-09-17T15:54:46.659 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36208.log 2024-09-17T15:54:46.659 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35898.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37383.log.gz 2024-09-17T15:54:46.659 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38874.log 2024-09-17T15:54:46.659 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36208.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35898.log.gz 2024-09-17T15:54:46.659 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39922.log 2024-09-17T15:54:46.660 INFO:teuthology.orchestra.run.smithi100.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.36208.log.gz 2024-09-17T15:54:46.660 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36993.log 2024-09-17T15:54:46.660 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39922.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39922.log.gz 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38874.log.gz 2024-09-17T15:54:46.660 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:54:46.660 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36779.log 2024-09-17T15:54:46.660 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36635.log 2024-09-17T15:54:46.660 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36779.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36993.log.gz 2024-09-17T15:54:46.660 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36243.log 2024-09-17T15:54:46.660 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36635.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36779.log.gz 2024-09-17T15:54:46.660 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-17T15:54:46.660 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36635.log.gz 2024-09-17T15:54:46.660 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39958.log 2024-09-17T15:54:46.661 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-mgr.x.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36243.log.gz 2024-09-17T15:54:46.661 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37099.log 2024-09-17T15:54:46.661 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39958.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39958.log.gz 2024-09-17T15:54:46.661 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37491.log 2024-09-17T15:54:46.661 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37099.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37099.log.gz 2024-09-17T15:54:46.661 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37420.log 2024-09-17T15:54:46.661 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37491.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37491.log.gz 2024-09-17T15:54:46.661 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35177.log 2024-09-17T15:54:46.661 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37420.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37420.log.gz 2024-09-17T15:54:46.661 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38085.log 2024-09-17T15:54:46.661 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35177.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35177.log.gz 2024-09-17T15:54:46.661 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39921.log 2024-09-17T15:54:46.661 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38085.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38085.log.gz 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr: 93.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32071.log 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35417.log 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.32071.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32071.log.gz 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39921.log.gz 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37062.log 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35829.log 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35417.log: /var/log/ceph/ceph-client.admin.37062.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35417.log.gz 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37062.log.gz 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35109.log 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36848.log 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35109.log: /var/log/ceph/ceph-client.admin.35829.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35109.log.gz 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36992.log 2024-09-17T15:54:46.662 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36848.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35829.log.gz 2024-09-17T15:54:46.663 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35418.log 2024-09-17T15:54:46.663 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36992.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36848.log.gz 2024-09-17T15:54:46.663 INFO:teuthology.orchestra.run.smithi100.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36992.log.gz 2024-09-17T15:54:46.663 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36414.log 2024-09-17T15:54:46.663 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36001.log 2024-09-17T15:54:46.663 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35418.log.gz/var/log/ceph/ceph-client.admin.36414.log: 2024-09-17T15:54:46.663 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37384.log 2024-09-17T15:54:46.663 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36001.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36414.log.gz 2024-09-17T15:54:46.663 INFO:teuthology.orchestra.run.smithi100.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36001.log.gz 2024-09-17T15:54:46.663 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38230.log 2024-09-17T15:54:46.663 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37847.log 2024-09-17T15:54:46.663 INFO:teuthology.orchestra.run.smithi100.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37384.log.gz/var/log/ceph/ceph-client.admin.38230.log: 2024-09-17T15:54:46.663 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35383.log 2024-09-17T15:54:46.664 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37847.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38230.log.gz 2024-09-17T15:54:46.664 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37847.log.gz 2024-09-17T15:54:46.664 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36344.log 2024-09-17T15:54:46.664 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35487.log 2024-09-17T15:54:46.664 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36344.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35383.log.gz 2024-09-17T15:54:46.664 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36140.log 2024-09-17T15:54:46.664 INFO:teuthology.orchestra.run.smithi100.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36344.log.gz 2024-09-17T15:54:46.664 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35487.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35487.log.gz 2024-09-17T15:54:46.664 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37169.log 2024-09-17T15:54:46.664 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37741.log 2024-09-17T15:54:46.664 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37169.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36140.log.gz 2024-09-17T15:54:46.665 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38193.log 2024-09-17T15:54:46.665 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37741.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37169.log.gz 2024-09-17T15:54:46.665 INFO:teuthology.orchestra.run.smithi100.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37741.log.gz 2024-09-17T15:54:46.665 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35281.log 2024-09-17T15:54:46.665 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38727.log 2024-09-17T15:54:46.665 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38193.log.gz 2024-09-17T15:54:46.665 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39479.log 2024-09-17T15:54:46.665 INFO:teuthology.orchestra.run.smithi100.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35281.log.gz 2024-09-17T15:54:46.665 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38337.log 2024-09-17T15:54:46.665 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39479.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38727.log.gz 2024-09-17T15:54:46.665 INFO:teuthology.orchestra.run.smithi100.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39479.log.gz 2024-09-17T15:54:46.665 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35933.log 2024-09-17T15:54:46.665 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40280.log 2024-09-17T15:54:46.666 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38337.log: /var/log/ceph/ceph-client.admin.35933.log: 87.8% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38337.log.gz 2024-09-17T15:54:46.666 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35933.log.gz 2024-09-17T15:54:46.666 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37633.log 2024-09-17T15:54:46.666 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37170.log 2024-09-17T15:54:46.666 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40280.log: /var/log/ceph/ceph-client.admin.37633.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40280.log.gz 2024-09-17T15:54:46.666 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35316.log 2024-09-17T15:54:46.666 INFO:teuthology.orchestra.run.smithi100.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37633.log.gz 2024-09-17T15:54:46.666 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37742.log 2024-09-17T15:54:46.666 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35316.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37170.log.gz 2024-09-17T15:54:46.666 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39264.log 2024-09-17T15:54:46.666 INFO:teuthology.orchestra.run.smithi100.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35316.log.gz 2024-09-17T15:54:46.666 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36105.log 2024-09-17T15:54:46.666 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39264.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37742.log.gz 2024-09-17T15:54:46.667 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37598.log 2024-09-17T15:54:46.667 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36105.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39264.log.gz 2024-09-17T15:54:46.667 INFO:teuthology.orchestra.run.smithi100.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35934.log 2024-09-17T15:54:46.667 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36105.log.gz 2024-09-17T15:54:46.667 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35314.log 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37598.log.gz 2024-09-17T15:54:46.667 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:54:46.667 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39372.log 2024-09-17T15:54:46.667 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35934.log: /var/log/ceph/ceph-client.admin.35314.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35934.log.gz 2024-09-17T15:54:46.667 INFO:teuthology.orchestra.run.smithi100.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35314.log.gz 2024-09-17T15:54:46.667 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38873.log 2024-09-17T15:54:46.667 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35075.log 2024-09-17T15:54:46.668 INFO:teuthology.orchestra.run.smithi100.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39372.log.gz/var/log/ceph/ceph-client.admin.38873.log: 2024-09-17T15:54:46.668 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39585.log 2024-09-17T15:54:46.668 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35075.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38873.log.gz 2024-09-17T15:54:46.668 INFO:teuthology.orchestra.run.smithi100.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35075.log.gz 2024-09-17T15:54:46.668 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38014.log 2024-09-17T15:54:46.668 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31991.log 2024-09-17T15:54:46.668 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38014.log: 88.5% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39585.log.gz 2024-09-17T15:54:46.668 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38014.log.gz 2024-09-17T15:54:46.668 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37918.log 2024-09-17T15:54:46.668 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.31991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36671.log 2024-09-17T15:54:46.668 INFO:teuthology.orchestra.run.smithi100.stderr: 88.6%/var/log/ceph/ceph-client.admin.37918.log: -- replaced with /var/log/ceph/ceph-client.admin.31991.log.gz 2024-09-17T15:54:46.668 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38123.log 2024-09-17T15:54:46.669 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37918.log.gz 2024-09-17T15:54:46.669 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36671.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36671.log.gz 2024-09-17T15:54:46.669 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36311.log 2024-09-17T15:54:46.669 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36956.log 2024-09-17T15:54:46.669 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36311.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38123.log.gz 2024-09-17T15:54:46.669 INFO:teuthology.orchestra.run.smithi100.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36311.log.gz 2024-09-17T15:54:46.669 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.f.log 2024-09-17T15:54:46.669 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36956.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36956.log.gz 2024-09-17T15:54:46.669 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39087.log 2024-09-17T15:54:46.669 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-mon.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38837.log 2024-09-17T15:54:46.669 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39087.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39087.log.gz 2024-09-17T15:54:46.669 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35726.log 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38837.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38837.log.gz 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39959.log 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35726.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35726.log.gz 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38514.log 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39959.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39959.log.gz 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38406.log 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38514.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38514.log.gz 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39407.log 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38406.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38406.log.gz 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37419.log 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39407.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39407.log.gz 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36037.log 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.37419.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37419.log.gz 2024-09-17T15:54:46.670 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-17T15:54:46.671 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36037.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36037.log.gz 2024-09-17T15:54:46.671 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40171.log 2024-09-17T15:54:46.671 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39086.log 2024-09-17T15:54:46.671 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40171.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40171.log.gz 2024-09-17T15:54:46.671 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38300.log 2024-09-17T15:54:46.671 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39086.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39086.log.gz 2024-09-17T15:54:46.671 INFO:teuthology.orchestra.run.smithi100.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.h.log.gz 2024-09-17T15:54:46.671 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35932.log 2024-09-17T15:54:46.671 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38300.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38300.log.gz 2024-09-17T15:54:46.671 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40064.log 2024-09-17T15:54:46.671 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35932.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35932.log.gz 2024-09-17T15:54:46.672 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36104.log 2024-09-17T15:54:46.672 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40064.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40064.log.gz 2024-09-17T15:54:46.672 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36207.log 2024-09-17T15:54:46.672 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36104.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36104.log.gz 2024-09-17T15:54:46.672 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40243.log 2024-09-17T15:54:46.672 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.36207.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36207.log.gz 2024-09-17T15:54:46.672 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39957.log 2024-09-17T15:54:46.672 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40243.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40243.log.gz 2024-09-17T15:54:46.672 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39957.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39957.log.gz 2024-09-17T15:54:46.672 INFO:teuthology.orchestra.run.smithi100.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-17T15:54:46.675 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42666.log 2024-09-17T15:54:46.675 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36877.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36877.log.gz 2024-09-17T15:54:46.675 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39818.log 2024-09-17T15:54:46.675 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38773.log 2024-09-17T15:54:46.675 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39818.log: 89.4% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42666.log.gz 2024-09-17T15:54:46.675 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41784.log 2024-09-17T15:54:46.676 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38773.log: -- replaced with /var/log/ceph/ceph-client.admin.39818.log.gz 2024-09-17T15:54:46.676 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40441.log 2024-09-17T15:54:46.676 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41784.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38773.log.gz 2024-09-17T15:54:46.677 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38254.log 2024-09-17T15:54:46.681 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40441.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41784.log.gz 2024-09-17T15:54:46.686 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37129.log 2024-09-17T15:54:46.690 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38254.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40441.log.gz 2024-09-17T15:54:46.695 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41115.log 2024-09-17T15:54:46.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37129.log: 90.0% -- replaced with /var/log/ceph/ceph-mon.d.log.gz 2024-09-17T15:54:46.697 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42372.log 2024-09-17T15:54:46.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41115.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41115.log.gz 2024-09-17T15:54:46.698 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37498.log 2024-09-17T15:54:46.699 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42372.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42372.log.gz 2024-09-17T15:54:46.699 INFO:teuthology.orchestra.run.smithi064.stderr:gzip 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38254.log.gz 2024-09-17T15:54:46.699 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42330.log 2024-09-17T15:54:46.700 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37498.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37498.log.gz 2024-09-17T15:54:46.700 INFO:teuthology.orchestra.run.smithi064.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37129.log.gz 2024-09-17T15:54:46.702 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42624.log 2024-09-17T15:54:46.702 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42330.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42330.log.gz 2024-09-17T15:54:46.702 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36172.log 2024-09-17T15:54:46.704 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42624.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42624.log.gz 2024-09-17T15:54:46.704 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40231.log 2024-09-17T15:54:46.706 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36172.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36172.log.gz 2024-09-17T15:54:46.706 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36919.log 2024-09-17T15:54:46.706 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40231.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40231.log.gz 2024-09-17T15:54:46.708 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36919.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36919.log.gz 2024-09-17T15:54:46.730 INFO:teuthology.orchestra.run.smithi100.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.i.log.gz 2024-09-17T15:54:46.797 INFO:teuthology.orchestra.run.smithi100.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.g.log.gz 2024-09-17T15:54:46.803 INFO:teuthology.orchestra.run.smithi064.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.e.log.gz 2024-09-17T15:54:46.850 INFO:teuthology.orchestra.run.smithi064.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-17T15:54:46.878 INFO:teuthology.orchestra.run.smithi064.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-17T15:54:47.063 INFO:teuthology.orchestra.run.smithi064.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-17T15:54:47.264 INFO:teuthology.orchestra.run.smithi100.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.f.log.gz 2024-09-17T15:54:48.674 INFO:teuthology.orchestra.run.smithi100.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-17T15:54:48.815 INFO:teuthology.orchestra.run.smithi100.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-17T15:54:49.056 INFO:teuthology.orchestra.run.smithi100.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-17T15:54:49.058 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-17T15:54:49.058 INFO:teuthology.orchestra.run.smithi100.stderr:real 0m2.662s 2024-09-17T15:54:49.058 INFO:teuthology.orchestra.run.smithi100.stderr:user 0m8.682s 2024-09-17T15:54:49.058 INFO:teuthology.orchestra.run.smithi100.stderr:sys 0m0.464s 2024-09-17T15:54:49.141 INFO:teuthology.orchestra.run.smithi064.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-17T15:54:49.217 INFO:teuthology.orchestra.run.smithi064.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-17T15:54:49.297 INFO:teuthology.orchestra.run.smithi064.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-17T15:54:49.299 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-17T15:54:49.299 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m2.924s 2024-09-17T15:54:49.299 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m10.349s 2024-09-17T15:54:49.299 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.398s 2024-09-17T15:54:49.299 INFO:tasks.ceph:Archiving logs... 2024-09-17T15:54:49.299 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/remote/smithi064/log 2024-09-17T15:54:49.301 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T15:54:51.047 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/remote/smithi100/log 2024-09-17T15:54:51.048 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T15:54:52.606 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T15:54:52.618 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T15:54:52.618 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T15:54:52.621 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T15:54:52.701 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T15:54:52.710 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T15:54:52.710 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:54:52.712 DEBUG:teuthology.orchestra.run.smithi100:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:54:52.728 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-09-17T15:54:52.730 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-09-17T15:54:52.733 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T15:54:52.733 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-09-17T15:54:52.733 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv01.front.sepia.ceph.com 3 6 377 18 -145us[ -131us] +/- 36ms 2024-09-17T15:54:52.733 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 16 +557us[ +557us] +/- 61ms 2024-09-17T15:54:52.733 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 -293us[ -293us] +/- 61ms 2024-09-17T15:54:52.733 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:54:52.735 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T15:54:52.735 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-09-17T15:54:52.735 INFO:teuthology.orchestra.run.smithi100.stdout:^* hv01.front.sepia.ceph.com 3 6 377 17 -144us[ -130us] +/- 36ms 2024-09-17T15:54:52.735 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 16 +540us[ +540us] +/- 61ms 2024-09-17T15:54:52.735 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 -298us[ -283us] +/- 61ms 2024-09-17T15:54:52.735 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:54:52.735 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T15:54:52.773 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T15:54:52.774 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T15:54:52.807 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T15:54:52.883 DEBUG:teuthology.orchestra.run.smithi100:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T15:54:52.953 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T15:54:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:54:52.985 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T15:54:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:54:53.015 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T15:54:53.024 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T15:54:53.049 INFO:teuthology.task.internal:Duration was 965.696714 seconds 2024-09-17T15:54:53.050 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T15:54:53.084 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T15:54:53.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T15:54:53.087 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T15:54:53.128 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T15:54:53.129 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T15:54:53.503 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T15:54:53.503 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-09-17T15:54:53.504 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T15:54:53.527 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T15:54:53.527 DEBUG:teuthology.orchestra.run.smithi100:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T15:54:53.554 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T15:54:53.554 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T15:54:53.570 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T15:54:53.623 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T15:54:53.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T15:54:53.640 DEBUG:teuthology.orchestra.run.smithi100:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T15:54:53.768 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T15:54:53.780 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T15:54:53.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T15:54:53.806 DEBUG:teuthology.orchestra.run.smithi100:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T15:54:53.832 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T15:54:53.842 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:54:53.850 DEBUG:teuthology.orchestra.run.smithi100:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:54:53.871 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-09-17T15:54:53.895 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = core 2024-09-17T15:54:53.927 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:54:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:54:53.946 DEBUG:teuthology.orchestra.run.smithi100:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:54:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:54:53.982 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T15:54:53.992 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T15:54:53.993 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/remote/smithi064 2024-09-17T15:54:53.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T15:54:54.027 DEBUG:teuthology.misc:Transferring archived files from smithi100:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908989/remote/smithi100 2024-09-17T15:54:54.027 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T15:54:54.060 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T15:54:54.060 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T15:54:54.066 DEBUG:teuthology.orchestra.run.smithi100:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T15:54:54.109 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T15:54:54.119 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T15:54:54.119 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T15:54:54.150 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T15:54:54.150 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T15:54:54.153 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T15:54:54.166 INFO:teuthology.orchestra.run.smithi064.stdout: 265951 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 15:54 /home/ubuntu/cephtest 2024-09-17T15:54:54.168 INFO:teuthology.orchestra.run.smithi100.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 15:54 /home/ubuntu/cephtest 2024-09-17T15:54:54.169 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T15:54:54.193 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T15:54:54.274 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete} duration: 965.696713924408 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-17T15:54:54.274 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T15:54:54.368 INFO:teuthology.run:pass